{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to train your own word vector embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Many tasks require embeddings or domain-specific vocabulary that pre-trained models based on a generic corpus may not represent well or at all. Standard word2vec models are not able to assign vectors to out-of-vocabulary words and instead use a default vector that reduces their predictive value.\n",
    "\n",
    "E.g., when working with industry-specific documents, the vocabulary or its usage may change over time as new technologies or products emerge. As a result, the embeddings need to evolve as well. In addition, corporate earnings releases use nuanced language not fully reflected in Glove vectors pre-trained on Wikipedia articles.\n",
    "\n",
    "We will illustrate the word2vec architecture using the keras library that we will introduce in more detail in the next chapter and the more performant gensim adaptation of the code provided by the word2vec authors. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate the word2vec network architecture, we use the TED talk dataset with aligned English and Spanish subtitles that we first introduced in chapter 13. \n",
    "\n",
    "This notebook contains the code to tokenize the documents and assign a unique id to each item in the vocabulary. We require at least five occurrences in the corpus and keep a vocabulary of 31,300 tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:40.090697Z",
     "start_time": "2018-12-10T06:52:39.318699Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from time import time\n",
    "from collections import Counter\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from numpy.linalg import norm\n",
    "from scipy.spatial.distance import cdist, cosine\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Dense, Reshape, Dot, Embedding\n",
    "from tensorflow.keras.preprocessing.sequence import skipgrams, make_sampling_table\n",
    "from tensorflow.keras.callbacks import Callback, TensorBoard\n",
    "\n",
    "from gensim.models import Word2Vec, KeyedVectors\n",
    "from gensim.models.word2vec import LineSentence\n",
    "from sklearn.decomposition import IncrementalPCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:40.096261Z",
     "start_time": "2018-12-10T06:52:40.091941Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "plt.style.use('ggplot')\n",
    "pd.set_option('float_format', '{:,.2f}'.format)\n",
    "%matplotlib inline\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:40.534840Z",
     "start_time": "2018-12-10T06:52:40.530045Z"
    }
   },
   "outputs": [],
   "source": [
    "PROJECT_DIR = Path().cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:45.285698Z",
     "start_time": "2018-12-10T06:52:45.283889Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "LANGUAGES = ['en', 'es']\n",
    "language_dict = dict(zip(LANGUAGES, ['English', 'Spanish']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## TED2013 Corpus Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inputs are produced by the [preprocessing](01_preprocessing.ipynb) notebook that needs to run first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:46.172123Z",
     "start_time": "2018-12-10T06:52:46.170203Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "SOURCE = 'TED'\n",
    "LANGUAGE = 'en'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:47.829019Z",
     "start_time": "2018-12-10T06:52:47.760797Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.io.pytables.HDFStore'>\n",
      "File path: vocab/TED/vocab.h5\n",
      "/en/tokens            frame        (shape->[3130266,3])\n",
      "/en/vocab             frame        (shape->[46176,1])  \n",
      "/es/tokens            frame        (shape->[2938546,3])\n",
      "/es/vocab             frame        (shape->[74739,1])  \n"
     ]
    }
   ],
   "source": [
    "with pd.HDFStore(Path('vocab', SOURCE, 'vocab.h5')) as store:\n",
    "    print(store.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:56.753472Z",
     "start_time": "2018-12-10T06:52:56.687010Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# words: 2,710,455\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_count</th>\n",
       "      <th>freq</th>\n",
       "      <th>n_words</th>\n",
       "      <th>vocab_size</th>\n",
       "      <th>coverage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>127548</td>\n",
       "      <td>1</td>\n",
       "      <td>127548</td>\n",
       "      <td>1</td>\n",
       "      <td>0.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>90702</td>\n",
       "      <td>1</td>\n",
       "      <td>90702</td>\n",
       "      <td>2</td>\n",
       "      <td>0.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>74494</td>\n",
       "      <td>1</td>\n",
       "      <td>74494</td>\n",
       "      <td>3</td>\n",
       "      <td>0.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>70274</td>\n",
       "      <td>1</td>\n",
       "      <td>70274</td>\n",
       "      <td>4</td>\n",
       "      <td>0.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>64387</td>\n",
       "      <td>1</td>\n",
       "      <td>64387</td>\n",
       "      <td>5</td>\n",
       "      <td>0.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>893</th>\n",
       "      <td>5</td>\n",
       "      <td>1755</td>\n",
       "      <td>8775</td>\n",
       "      <td>17026</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>894</th>\n",
       "      <td>4</td>\n",
       "      <td>2484</td>\n",
       "      <td>9936</td>\n",
       "      <td>19510</td>\n",
       "      <td>0.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>895</th>\n",
       "      <td>3</td>\n",
       "      <td>3831</td>\n",
       "      <td>11493</td>\n",
       "      <td>23341</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>896</th>\n",
       "      <td>2</td>\n",
       "      <td>6822</td>\n",
       "      <td>13644</td>\n",
       "      <td>30163</td>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>897</th>\n",
       "      <td>1</td>\n",
       "      <td>16013</td>\n",
       "      <td>16013</td>\n",
       "      <td>46176</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     word_count   freq  n_words  vocab_size  coverage\n",
       "0        127548      1   127548           1      0.05\n",
       "1         90702      1    90702           2      0.08\n",
       "2         74494      1    74494           3      0.11\n",
       "3         70274      1    70274           4      0.13\n",
       "4         64387      1    64387           5      0.16\n",
       "893           5   1755     8775       17026      0.98\n",
       "894           4   2484     9936       19510      0.98\n",
       "895           3   3831    11493       23341      0.99\n",
       "896           2   6822    13644       30163      0.99\n",
       "897           1  16013    16013       46176      1.00"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with pd.HDFStore(Path('vocab', SOURCE, 'vocab.h5')) as store:\n",
    "    df = store['{}/vocab'.format(LANGUAGE)]\n",
    "\n",
    "wc = df['count'].value_counts().sort_index(ascending=False).reset_index()\n",
    "wc.columns = ['word_count', 'freq']\n",
    "wc['n_words'] = wc.word_count.mul(wc.freq)\n",
    "\n",
    "wc['corpus_share'] = wc.n_words.div(wc.n_words.sum())\n",
    "wc['coverage'] = wc.corpus_share.cumsum()\n",
    "wc['vocab_size'] = wc.freq.cumsum()\n",
    "\n",
    "print('# words: {:,d}'.format(wc.n_words.sum()))\n",
    "(wc\n",
    " .loc[:, ['word_count', 'freq', 'n_words', 'vocab_size', 'coverage']]\n",
    " .head()\n",
    " .append(wc\n",
    "         .loc[:, ['word_count', 'freq', 'n_words', 'vocab_size', 'coverage']]\n",
    "         .tail()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:52:58.961174Z",
     "start_time": "2018-12-10T06:52:58.951088Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count   898.00\n",
       "mean      0.09\n",
       "std       0.30\n",
       "min       0.00\n",
       "50%       0.02\n",
       "75%       0.04\n",
       "95%       0.31\n",
       "96%       0.37\n",
       "97%       0.49\n",
       "98%       0.62\n",
       "99%       1.63\n",
       "99.9%     3.49\n",
       "max       4.71\n",
       "Name: word_count, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wc.word_count.div(wc.n_words.sum()).mul(100).describe(percentiles=[.5, .75, .95, .96, .97, .98, .99, .999])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Evaluation: Analogies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dimensions of the word and phrase vectors do not have an explicit meaning. However, the embeddings encode similar usage as proximity in the latent space in a way that carries over to semantic relationships. This results in the interesting properties that analogies can be expressed by adding and subtracting word vectors.\n",
    "\n",
    "Just as words can be used in different contexts, they can be related to other words in different ways, and these relationships correspond to different directions in the latent space. Accordingly, there are several types of analogies that the embeddings should reflect if the training data permits.\n",
    "\n",
    "The word2vec authors provide a list of several thousand relationships spanning aspects of geography, grammar and syntax, and family relationships to evaluate the quality of embedding vectors (see directory [analogies](data/analogies))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:02.611652Z",
     "start_time": "2018-12-10T06:53:02.535004Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(Path('data', 'analogies', 'analogies-en.txt'), header=None, names=['category'], squeeze=True)\n",
    "categories = df[df.str.startswith(':')]\n",
    "analogies = df[~df.str.startswith(':')].str.split(expand=True)\n",
    "analogies.columns = list('abcd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:02.750899Z",
     "start_time": "2018-12-10T06:53:02.724987Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>: capital-common-countries</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>baghdad</td>\n",
       "      <td>iraq</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>: capital-common-countries</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>bangkok</td>\n",
       "      <td>thailand</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>: capital-common-countries</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>beijing</td>\n",
       "      <td>china</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>: capital-common-countries</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>berlin</td>\n",
       "      <td>germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>: capital-common-countries</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>bern</td>\n",
       "      <td>switzerland</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       category       a       b        c            d\n",
       "1  : capital-common-countries    athens  greece  baghdad         iraq\n",
       "2  : capital-common-countries    athens  greece  bangkok     thailand\n",
       "3  : capital-common-countries    athens  greece  beijing        china\n",
       "4  : capital-common-countries    athens  greece   berlin      germany\n",
       "5  : capital-common-countries    athens  greece     bern  switzerland"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.concat([categories, analogies], axis=1)\n",
    "df.category = df.category.ffill()\n",
    "df = df[df['a'].notnull()]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:03.107617Z",
     "start_time": "2018-12-10T06:53:03.068779Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "analogy_cnt = df.groupby('category').size().sort_values(ascending=False).to_frame('n')\n",
    "analogy_example = df.groupby('category').first()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:05.515262Z",
     "start_time": "2018-12-10T06:53:05.500372Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n</th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>: capital-world</th>\n",
       "      <td>8556</td>\n",
       "      <td>abuja</td>\n",
       "      <td>nigeria</td>\n",
       "      <td>accra</td>\n",
       "      <td>ghana</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: city-in-state</th>\n",
       "      <td>4242</td>\n",
       "      <td>chicago</td>\n",
       "      <td>illinois</td>\n",
       "      <td>houston</td>\n",
       "      <td>texas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram6-nationality-adjective</th>\n",
       "      <td>1640</td>\n",
       "      <td>albania</td>\n",
       "      <td>albanian</td>\n",
       "      <td>argentina</td>\n",
       "      <td>argentinean</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram7-past-tense</th>\n",
       "      <td>1560</td>\n",
       "      <td>dancing</td>\n",
       "      <td>danced</td>\n",
       "      <td>decreasing</td>\n",
       "      <td>decreased</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram8-plural</th>\n",
       "      <td>1332</td>\n",
       "      <td>banana</td>\n",
       "      <td>bananas</td>\n",
       "      <td>bird</td>\n",
       "      <td>birds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram3-comparative</th>\n",
       "      <td>1332</td>\n",
       "      <td>bad</td>\n",
       "      <td>worse</td>\n",
       "      <td>big</td>\n",
       "      <td>bigger</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram4-superlative</th>\n",
       "      <td>1122</td>\n",
       "      <td>bad</td>\n",
       "      <td>worst</td>\n",
       "      <td>big</td>\n",
       "      <td>biggest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram5-present-participle</th>\n",
       "      <td>1056</td>\n",
       "      <td>code</td>\n",
       "      <td>coding</td>\n",
       "      <td>dance</td>\n",
       "      <td>dancing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram1-adjective-to-adverb</th>\n",
       "      <td>992</td>\n",
       "      <td>amazing</td>\n",
       "      <td>amazingly</td>\n",
       "      <td>apparent</td>\n",
       "      <td>apparently</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram9-plural-verbs</th>\n",
       "      <td>870</td>\n",
       "      <td>decrease</td>\n",
       "      <td>decreases</td>\n",
       "      <td>describe</td>\n",
       "      <td>describes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: currency</th>\n",
       "      <td>866</td>\n",
       "      <td>algeria</td>\n",
       "      <td>dinar</td>\n",
       "      <td>angola</td>\n",
       "      <td>kwanza</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: gram2-opposite</th>\n",
       "      <td>812</td>\n",
       "      <td>acceptable</td>\n",
       "      <td>unacceptable</td>\n",
       "      <td>aware</td>\n",
       "      <td>unaware</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: family</th>\n",
       "      <td>506</td>\n",
       "      <td>boy</td>\n",
       "      <td>girl</td>\n",
       "      <td>brother</td>\n",
       "      <td>sister</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>: capital-common-countries</th>\n",
       "      <td>506</td>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>baghdad</td>\n",
       "      <td>iraq</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    n           a             b           c  \\\n",
       "category                                                                      \n",
       ": capital-world                  8556       abuja       nigeria       accra   \n",
       ": city-in-state                  4242     chicago      illinois     houston   \n",
       ": gram6-nationality-adjective    1640     albania      albanian   argentina   \n",
       ": gram7-past-tense               1560     dancing        danced  decreasing   \n",
       ": gram8-plural                   1332      banana       bananas        bird   \n",
       ": gram3-comparative              1332         bad         worse         big   \n",
       ": gram4-superlative              1122         bad         worst         big   \n",
       ": gram5-present-participle       1056        code        coding       dance   \n",
       ": gram1-adjective-to-adverb       992     amazing     amazingly    apparent   \n",
       ": gram9-plural-verbs              870    decrease     decreases    describe   \n",
       ": currency                        866     algeria         dinar      angola   \n",
       ": gram2-opposite                  812  acceptable  unacceptable       aware   \n",
       ": family                          506         boy          girl     brother   \n",
       ": capital-common-countries        506      athens        greece     baghdad   \n",
       "\n",
       "                                           d  \n",
       "category                                      \n",
       ": capital-world                        ghana  \n",
       ": city-in-state                        texas  \n",
       ": gram6-nationality-adjective    argentinean  \n",
       ": gram7-past-tense                 decreased  \n",
       ": gram8-plural                         birds  \n",
       ": gram3-comparative                   bigger  \n",
       ": gram4-superlative                  biggest  \n",
       ": gram5-present-participle           dancing  \n",
       ": gram1-adjective-to-adverb       apparently  \n",
       ": gram9-plural-verbs               describes  \n",
       ": currency                            kwanza  \n",
       ": gram2-opposite                     unaware  \n",
       ": family                              sister  \n",
       ": capital-common-countries              iraq  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogy_cnt.join(analogy_example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `word2vec` - skipgram Architecture using Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:11.962382Z",
     "start_time": "2018-12-10T06:53:11.957411Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "NGRAMS = 3                                # Longest ngram in text\n",
    "FILE_NAME = 'ngrams_{}'.format(NGRAMS)    # Input to use\n",
    "MIN_FREQ = 5\n",
    "SAMPLING_FACTOR = 1e-4\n",
    "WINDOW_SIZE = 5\n",
    "EMBEDDING_SIZE = 200\n",
    "EPOCHS = 1\n",
    "BATCH_SIZE = 50\n",
    "\n",
    "# Set up validation\n",
    "VALID_SET = 10      # Random set of words to get nearest neighbors for\n",
    "VALID_WINDOW = 150  # Most frequent words to draw validation set from\n",
    "NN = 10             # Number of nearest neighbors for evaluation\n",
    "\n",
    "valid_examples = np.random.choice(VALID_WINDOW, size=VALID_SET, replace=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:53:13.843939Z",
     "start_time": "2018-12-10T06:53:13.841581Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "path = Path('keras', SOURCE, LANGUAGE, FILE_NAME).resolve()\n",
    "tb_path = path / 'tensorboard'\n",
    "if not tb_path.exists():\n",
    "    tb_path.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Build Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Tokens to ID\n",
    "\n",
    "1. Extract the top *n* most common words to learn embeddings\n",
    "2. Index these *n* words with unique integers\n",
    "3. Create an `{index: word}` dictionary\n",
    "4. Replace the *n* words with their index, and a dummy value `UNK` elsewhere"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:02.709606Z",
     "start_time": "2018-12-10T06:54:02.705898Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def build_data(language, ngrams=1):\n",
    "    file_path = PROJECT_DIR / 'vocab' / SOURCE / language / 'ngrams_{}.txt'.format(ngrams)\n",
    "    words = file_path.read_text().split()\n",
    "    \n",
    "    # Get (token, count) tuples for tokens meeting MIN_FREQ \n",
    "    token_counts = [t for t in Counter(words).most_common() if t[1] >= MIN_FREQ]\n",
    "    tokens, counts = list(zip(*token_counts))\n",
    "    \n",
    "    # create id-token dicts & reverse dicts\n",
    "    id_to_token = pd.Series(tokens, index=range(1, len(tokens) + 1)).to_dict()\n",
    "    id_to_token.update({0: 'UNK'})\n",
    "    token_to_id = {t:i for i, t in id_to_token.items()}\n",
    "    data = [token_to_id.get(word, 0) for word in words]\n",
    "    return data, token_to_id, id_to_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:05.230482Z",
     "start_time": "2018-12-10T06:54:03.089009Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data, token_to_id, id_to_token = build_data(LANGUAGE, ngrams=NGRAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:07.383676Z",
     "start_time": "2018-12-10T06:54:07.381927Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "vocab_size = len(token_to_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:07.562714Z",
     "start_time": "2018-12-10T06:54:07.558075Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17381"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:14.901599Z",
     "start_time": "2018-12-10T06:54:14.745194Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 17380)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(data), max(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:18.499601Z",
     "start_time": "2018-12-10T06:54:17.631605Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "s = pd.Series(data).value_counts().reset_index()\n",
    "s.columns = ['id', 'count']\n",
    "s['token'] = s.id.map(id_to_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:18.508607Z",
     "start_time": "2018-12-10T06:54:18.500964Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>count</th>\n",
       "      <th>token</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>127548</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>90702</td>\n",
       "      <td>and</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>74494</td>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70274</td>\n",
       "      <td>of</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>64387</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>56578</td>\n",
       "      <td>that</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>51254</td>\n",
       "      <td>UNK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>49621</td>\n",
       "      <td>i</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>47404</td>\n",
       "      <td>in</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>46931</td>\n",
       "      <td>it</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id   count token\n",
       "0   1  127548   the\n",
       "1   2   90702   and\n",
       "2   3   74494    to\n",
       "3   4   70274    of\n",
       "4   5   64387     a\n",
       "5   6   56578  that\n",
       "6   0   51254   UNK\n",
       "7   7   49621     i\n",
       "8   8   47404    in\n",
       "9   9   46931    it"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.sort_values('count', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:18.551156Z",
     "start_time": "2018-12-10T06:54:18.509623Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: The signature of `Series.to_csv` was aligned to that of `DataFrame.to_csv`, and argument 'header' will change its default value from False to True: please pass an explicit value to suppress this warning.\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "s.sort_values('id').token.dropna().to_csv(tb_path / 'meta.tsv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Analogies to ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:23.842956Z",
     "start_time": "2018-12-10T06:54:23.690010Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(Path('data', 'analogies', 'analogies-{}.txt'.format(LANGUAGE)), \n",
    "                 header=None, squeeze=True)\n",
    "categories = df[df.str.startswith(':')]\n",
    "analogies = df[~df.str.startswith(':')].str.split(expand=True)\n",
    "analogies.columns = list('abcd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:24.456585Z",
     "start_time": "2018-12-10T06:54:24.436606Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>baghdad</td>\n",
       "      <td>iraq</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>bangkok</td>\n",
       "      <td>thailand</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>beijing</td>\n",
       "      <td>china</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>berlin</td>\n",
       "      <td>germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>athens</td>\n",
       "      <td>greece</td>\n",
       "      <td>bern</td>\n",
       "      <td>switzerland</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        a       b        c            d\n",
       "1  athens  greece  baghdad         iraq\n",
       "2  athens  greece  bangkok     thailand\n",
       "3  athens  greece  beijing        china\n",
       "4  athens  greece   berlin      germany\n",
       "5  athens  greece     bern  switzerland"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:54:26.357446Z",
     "start_time": "2018-12-10T06:54:26.263768Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.332703213610586"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "analogies_id = analogies.apply(lambda x: x.map(token_to_id))\n",
    "analogies_id.notnull().all(1).sum()/len(analogies_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Generate Sampling Probabilities\n",
    "\n",
    "There is an alternative, faster scheme than the traditional SoftMax loss function called [Noise Contrastive Estimation (NCE)](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf).\n",
    "\n",
    "Instead of getting the softmax probability for all possible context words, randomly sample 2-20 possible context words and evaluate the probability only for these."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**SAMPLING_FACTOR**: used for generating the `sampling_table` argument for `skipgrams`. \n",
    "\n",
    "`sampling_table[i]` is the probability of sampling the word i-th most common word in a dataset\n",
    "\n",
    "The sampling probabilities are generated according\n",
    "to the sampling distribution used in word2vec:\n",
    "\n",
    "$p(\\text{word}) = \\min(1, \\frac{\\sqrt{\\frac{\\text{word frequency}}{\\text{sampling factor}}}}{\\frac{\\text{word frequency}}{\\text{sampling factor}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:57:06.462506Z",
     "start_time": "2018-12-10T06:57:05.855693Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8leWd///X2XKSnGyQsCTsSwhGdkSoKW4IgsVtRi8BZ7T9Wq226kz7bTt1nHYej371VzvtzPy0Vvpl6taq4KVtUawKarFUBdlkkX0JS4AEQkKWk+ScnOX7xzk5JCQh21muhM/z8eAB5z738s5N8sl1rvu6r9sSDAYRQgjRd1kTHUAIIURsSaEXQog+Tgq9EEL0cVLohRCij5NCL4QQfZwUeiGE6OOk0AshRB8nhV4IIfo4KfRCCNHH2RMdIExuzxVCiO6xdLSCKYWekydPJjpCm3JycigvL090jFZMzQXmZjM1F5ibzdRcYG62eObKy8vr1HrSdSOEEH2cFHohhOjjpNALIUQfl9A+eqXUzcDNWutExhBCGCIYDNLQ0EAgEMBiufg1xrKyMjweT5ySdV60cwWDQaxWK8nJyR2ek/YktNBrrVcBq4D7E5lDCGGGhoYGHA4HdnvHpclut2Oz2eKQqmtikcvn89HQ0EBKSkq3tpeuGyGEMQKBQKeK/KXGbrcTCAS6vb0UeiGEMbrbNXEp6Mm5kUIvhBB9nDGFPmjgRRUhxKVp7dq1zJ49m6KiIp599tlW73s8Hh588EGKiopYuHAhx48fB6CiooLbb7+d/Px8Hn/88XjHbpcxhV5mQRBCmMDv9/P444/zyiuvsHbtWlauXMn+/ftbrLN8+XIyMzP59NNPuf/++3nyyScBSE5O5kc/+hE//vGPExG9XeYU+mD3LzQIIUS0fPHFF4wcOZIRI0aQlJTErbfeyurVq1uss2bNGu68804Avva1r/HJJ58QDAZJTU1l5syZOJ3ORERvlzmXt3twRVkI0fcEVvwPwePF7b9vsRAMdq0nwDJsFNZFFx/NXVpa2mIOmdzcXL744ot217Hb7WRkZFBZWUn//v27lCdezGnRS6EXQhigrV8eF4546eovmEQz587YXnbihBCx1VHL22634/P5on7c3NzcFrPpnjp1ikGDBrW5Tl5eHj6fj+rqavr16xf1LNGS0Ba91nqV1voBQProhRBGmDJlCsXFxRw7dgyv18tbb73FvHnzWqwzb9483njjDQD+/Oc/U1RUZPQ9ANJHL4QQzdjtdp544gmWLFlCIBDgrrvuoqCggF/84hdMnjyZefPmsWjRIh599FGKiorIysriueeei2x/xRVXUFNTg9fr5f3332f58uWMGzcugV8RWAzpawqe+HIHlv45ic7RijzcoOtMzWZqLjA3W7xz1dXVkZqa2ql1Y9V101OxytXWuQlfEO7wo4Q5F2Ol60YIIWLCnEIvXTdCCBETxhT64I5NiY4ghBB9kjGFniMHE51ACCH6JHMK/YDBiU4ghBB9kjmFPndYohMIIUSfZE6hb/QmOoEQQgDdn6YY4Omnn6aoqIjZs2fz8ccfR5Z/73vfY9KkSVx//fXx+BJaMKfQ+xoTnUAIIXo0TfH+/ftZuXIlf/nLX3j11Vf513/9V/x+PwBKKV599dW4fz0Qo0KvlHIppbYopRZ2eqNGKfRCiMTryTTFq1ev5rbbbsPpdDJ8+HBGjhwZmfly1qxZZGVlxf3rgU5OgaCUegFYCJzWWk9otnw+8DRgA36rtX4q/Na/ALpLSbzyhCkhxHm/3VxGcWVDu+9bujFN8ah+yXzzikEXXacn0xSXlpYyY8aMFtuWlpZ2KWMsdHaum5eAZ4HfNS1QStmAXwNzgRJgk1LqbSAP2A0kdymJp75LqwshRCz0ZJrizmybCJ0q9FrrdUqpkRcsvhI4qLU+DKCUWgHcCqQBLqAQqFdKvau17vi2V3lmrBCimY5a3iZOU5ybm8uJEycuum0i9GT2yiHA8WavS4CZWuuHAZRSXwfK2yvySqkHgAcAtNYkWyEjx7xJzex2OzmSq0tMzWZqLjA3W7xzlZWVYbd3vix1Zd3OuuKKKyguLubEiRPk5uby9ttvs3Tp0hbHmj9/Pn/4wx+YNWsWq1atYvbs2TgcDhYsWMBDDz3Egw8+SGlpKcXFxcyYMQObzQYQ+bs7uZ1OZ7f/L3pyltr6PBL53KK1fuliG2utlwHLmrZrqDqHV2bv6zRTc4G52UzNBeZmi3cuj8cTKYYdieXslU888QR33XVXZJrisWPH8rOf/SwyTbFSikcffZSZM2dGpin2+XyMHTuWW265hdmzZ2Oz2XjyyScJBoP4fD6+/e1vs379eioqKpg8eTLf//73Wbx4caczeTyeVv8Xza8lXExPCn0J0Pwup6HAyXbWbVPzJ0wFPe1fdBFCiHiaM2cOc+bMabHsBz/4QeTfycnJLFu27MLNAPjud7/LI4880mp58znr460nhX4TkK+UGgWcABYBS7qyA631KmAVcD9S6IUQIiY6NY5eKbUcWA8UKKVKlFL3aa19wMPAamAPoLXWu7qdRIZXCiFETHR21E2bHUla63eBd7t7cOm6EUKI2EvoM2Obd900NPpJS2QYIYToo4yZ68YiLXohhIiJhLbom3fdBLwy140QQsRCQlv0WutVWusHAAKN3i7PWyGEELEQi2mK29vniy++SFFREUOGDKGioiImX48xXTfBIDInvRAi4WIxTfHF9jljxgxWrFjB0KFDY/Y1GVPo/RYLMpZeCJFosZim+GL7nDBhAsOGxfYJe8b00QctFqiugvTMREYSQhjiy611VJ/zt/t+d6YpzsiyMWFa6kXXidU0xR3tM5aMGV4ZxAoNdYmMI4QQMZmmOBBoPbdjPKcvTmihby5gsUCDzEkvhAjpqOXd26Yp7mifsWRMH30Ai7TohRAJN2XKFIqLizl27Bher5e33nqLefPmtVhn3rx5vPHGGwD8+c9/pqioCIvFwrx581i5ciUej4djx45RXFzM1KlTO7XPWDKmjz5gsRJsqG9z7mMhhIgXu93OE088wZIlSyLTFBcUFPCLX/wiMk3xokWLePTRRykqKopMUwxQUFDALbfcwnXXXReZprhp2uW29gnw/PPP89xzz3HmzBluuOEGrr/+en75y19G9WuyGDJ2Pfizh/+Nf5w5DOucmxOdpQWZJ7zrTM1mai4wN1u8c9XV1ZGaevEumyaxnI++J2KVq61zE77A22H72Jium/LkLKhzJzqGEEL0OcYUeqvFCnW1iY4hhBB9jjGF/tOcCdKiF0KIGDDmYuw11fsI2mR4pRBCRJsxN0x9mHU53y7fmcg4QgjRJxnTdQNAvYyjF0KIaDOq0Ne6ZVIzIURife9732PSpElcf/31Xd52x44dXHPNNRQVFfHjH/84MiXCf/7nfzJ9+nTmzp3L3Llz+eijj6Id+6KMKvQ7HAMTHUEIcYlTSvHqq692a9vHHnuMX/7yl3zyyScUFxezdu3ayHv3338/H3zwAR988AFz5syJVtxOMarQ73UOloePCCESatasWWRlZbVYduTIEe6++27mz5/P7bffzsGDB1ttV1ZWRk1NDTNmzMBisXDHHXfw/vvvxyv2RRkzqRlAo8UGnnpI7tydcUKIvmvdunWcOXOm3fe7M03xgAEDuPrqq7uc5Yc//CFPPfUUo0ePZuvWrTz22GORuW6alJaWkpubG3ndfIpiCD1J6s0332TSpEn85Cc/afXLJJaMGV6ZZg0AwdCc9FLohRCGcLvdbNmyhW9961uRZV5v66fhXWx643vuuYd//ud/xmKx8B//8R/89Kc/5b/+679iF/oCxgyvzEmCs84sqK6EgbkdbSqE6OM6annHa66bQCBARkYGH3zwQYvlfr+f+fPnA6HZLO+55x5OnToVeb/5VMQDBgyILL/77ru59957Y567OWP66LNTbJx1ZkL1uURHEUKIiPT0dIYNG8aqVauAUMt9165d2Gy2yMXVH/zgBwwaNIi0tDQ2b95MMBjkzTff5MYbbwRC/fdN3nvvvcjMlfFiTB99tsvBAWcmwfoqmapYCJEw3/72t1m/fj0VFRVMnz6d73//+zz77LM89thjPP300/h8Pm699VYuv/zyVtv+7Gc/47vf/S4NDQ1cd911kSGaTzzxBLt378ZisTB06FB+/vOfx/VrMqfQpzqpTkrDW19GcqLDCCEuWU1zy1+oM0MuJ0+ezLp161p1Kf3qV7+KSrbuMqfrJi1U3s+6W1/kEEII0X3GFPqcDCcAZxraf+q7EEKIrjOm0GcnOwCokAa9EJcsuWGyfT05N8YU+qzk0OWCikZjIgkh4sxqtRr5eMBE8/l8WK3dr43G3DDltFtIDXip8NsSGUkIkUDJyck0NDTg8XgiNxu1x+l04vF44pSs86KdKxgMYrVaSU7u/jAVY26YIgj9AvWcDSYlMpIQIoEsFgspKSmdWlceqN55xvSTBIOQbfFSaXEmOooQQvQpRhX6/vYAFdbO/TYXQgjROcYUeoBgkpMzyf3w1dYmOooQQvQZxhT6YDBIMCl0seHo8bIO1hZCCNFZxhR6vw9uGuUCoPx0RYLTCCFE32FMoS8+4GHo8ND0xCfLaxKcRggh+g5jCr2vMUh6ZhouXwOlbrlhQgghosWY2Ss9ntDtvYMCtZwKGBNLCCF6PWNa9PXuAAB5Ni+nrPIoQSGEiBZjCr0rPRQlL9lCuSMDr0dmNxNCiGgwptCfPhXql8/tl0LAYqXsaEmCEwkhRN8Q9c5wpdRlwD8BOcBHWuulXdl+xNABcLqRw8fOMGzc6GjHE0KIS06nCr1S6gVgIXBaaz2h2fL5wNOADfit1voprfUe4EGllBX4n66Ecdf6GTZyCNYthzl+rr4rmwohhGhHZ7tuXgLmN1+glLIBvwYWAIXAYqVUYfi9W4BPgI+6Eqaq0k9SSjKDvFWUhC/OCiGE6JlOFXqt9TrgwttVrwQOaq0Pa629wArg1vD6b2utrwLu7kqY0pJGAPJtdeygH355AIEQQvRYT/rohwDHm70uAWYqpa4F/g5wAu+2t7FS6gHgAQCtNQD9+rvIyclh2ohs1p1IprGqlkEFY3sQsefsdjs5OTkJzdAWU3OBudlMzQXmZjM1F5ibzcRcPSn0bT3+Jai1/hj4uKONtdbLgGVN2zmTLVRX11FeXs7ATCecgJ17i0nOzupBxJ4z8SECYG4uMDebqbnA3Gym5gJzs8UzV15eXqfW68nwyhJgWLPXQ4GTXdmBUupmpdQygOQUK/V14Zum8gYAUFYp0xULIURP9aRFvwnIV0qNAk4Ai4AlXdlB80cJpqRacdf4Acgc0J803zH2nfWysAcBhRBCdLJFr5RaDqwHCpRSJUqp+7TWPuBhYDWwB9Ba613dDZKcYqG+PtSit1qtzLRWsIlsuUNWCCF6qFMteq314naWv8tFLrh2RCl1M3Cz1hpnMvgaQ7NY2h0Wpg/N5KOTyRzafYjLpl7W3UMIIcQlL6FTIGitV2mtHwBIcoZmr2zqp580pQCn38vHu7vU7S+EEOICxsx1k5waitJU6NP7ZXC5v5ztnlQCAbl5Sgghuiuhhb7FqJvkUIu+of58UZ8+wMEpZz+K9x5OTEAhhOgDjOm6cThDw/Ib6oOR92dfNRFr0M/fdhxLTEAhhOgDjOm6sVrBkWRp0aLP7J/FlMYyPm5Ip9HbmMB0QgjRexnTdRMMBklJaVnoARaMzaLSkc7mz7YnJKMQQvR2CX04a/MbpoLBIM4Ua4uuG4BpsyaRdWgbHx+p4isJSSmEEL2bMV03wWCQ5BRrqxa93WGnyFHFZtsgKs+cTVA6IYTovYwq9ACehiCN3pat+vkzx+Kz2nn7L9J9I4QQXWVUoW8q9iePt5z2YPjYEeR5KtnqTkpENCGE6NWMuRhbU1PDZZNSADhX4W+17rWZHo44c6T7RgghusiYcfS7du0iOcVKZj8b1edaF/qJowcC8OXOQ/ENKYQQvZwxXTdNUl1WzlX4W12UHTdxHNneaj484k5QMiGE6J2MKfQnT4YmLxsxNtQPv3t7fYv37XY7c9PcbHPmUnr0RNzzCSFEb2VMoe/fvz8AOQNDQ/tPHG3E62nZqr/hqkKswQBrNuyLez4hhOitjLkYW1VVBYDFYqFgQjIAlRdclB0wZBDTGkv5qD5DpkQQQohOMuZibGVlZWT5yPxQ983Gda374+fnZ3HOkcZfPtoYp5RCCNG7GdN1A+dvmkpKOh9r86cti/3UWZMY7TnDm6V2fI2+uOYTQojeyKhCX15eHvn3TXdkAnCqpJH9uxoiy+12O3eMSuZ0Uiab12+Le0YhhOhtjC30NpuFr85JA2Dflw3s332+2M/86lRyvNX830NBas5Vxz2nEEL0JkYV+kOHWt4M1S/HzpyvpQOwb2cDZ0+HumrsDjvfn5JORVI6r/x5U9xzCiFEb2JUoe/Xr1+rZalpNorCLfvP1tZGll829TKme0/yPkPkUYNCCHERxgyvTEtLw+1u+67X/jl28gudAJw4dn7Cs4dvmoQt4OeFz47i88mFWSGEaIsxwyvT0tKora1td90xBaGx9VvX10VG5/QflMM3MivY4czlLx/KcEshhGiLMV036enpFy30jiQL/QfYAHj/T1WR5QsWfIVRnjP8vtSJu7r97YUQ4lJlTKFvatE3tdbbctV1ob56XyN8vi5U1O0OOw9OG0C1w8WLb22IS1YhhOhNjCr0Pp8Pj8fT7joWi4Wv3RkaX3/6lI9Vr5/D7w8yfsp4buIEH1iH8s47f4tXZCGE6BWMKvQQegDJxVitFhb8fWbk9btvVlFT7ec+dQ2TPKd4obI/h3YfjGlWIYToTYwp9OnpofHyHRV6ALvdwkKVydCRDgA+fq+GY8V+Hro+H5e/gZ9/Xk51ZVUHexFCiEuDMYW+qUXf3hDLC1ksFqbOdJHVP3SBdtcX9RzYm8EPJzg548jg3/+0g3p3XczyCiFEb2FMoU9NTcVisXS60DeZPTedGV91AVBdFeDIoaHcl+bnsHMAT73+OYFAoIM9CCFE32bMDVNWq5XU1NSLDrFsz+AhDm76+0xyh4a6coKeYXzTPpjjzmGsWvVJdEMLIUQvY0/kwbXWq4BVwP0ALperyy36Jja7hSuKXJwq8bL501CXze32HD5y2xm/fS8Fk8dHK7YQQvQqxnTdQM8KfZPcoUncfFcWYy8LTZkwx5bF/r2D2fH5gWhEFEKIXseoQt/RNAhdcdmkFK4oSo28PnpkACeOnI7KvoUQojcxqtB7PB4aGhqoq4vOaJmm1r3FUQbA5xvsuGtlJI4Q4tJiVKFvGmK5a9euqO534d8VUBs4jcNi5U9vn6XB3f7dt0II0dcYVehnzpwJQFVV9G92Wrx4HHXBUjJt6bz5VmnU9y+EEKYyqtA7HA7y8vLYvXt3TPZ/16LxVPmqybRl8sbre2JyDCGEMI1RhR7OT4VQWVkZk/3ftjCb+kAjyeRKsRdCXBKMK/RN3TcrV66Myf7TMl3ccEMS1X43yeSyYvk+/H5/TI4lhBAmMK7QZ2VlAaHJzaqrq2NyjP4DMrnz7wdS5a/BZR3E8tdPUXW248nUhBCiNzKu0AMsWLAAgNOnYzfuPcnpYPFdedT4z5BpS2PtBz4++8u+mB1PCCESJSaFXil1m1Lqf5RSbyml5nV1+5EjRwLw7rvv0tjYGO14ETabjSVL8klJP4kVOHtmEPr1vTE7nhBCJEKnC71S6gWl1Gml1JcXLJ+vlNqnlDqolPoRgNZ6pdb6fuDrwF1dDeVwOCJj6pcuXdrVzbvshpsKmXV1kNqAhxQG88prR6UrRwjRZ3SlRf8SML/5AqWUDfg1sAAoBBYrpQqbrfJv4fe77Bvf+AYuV2j64ddee+2iz5KNhoF5/bnjjv5U+avItGWy7kM/r6+QC7VCiN6v04Vea70OqLhg8ZXAQa31Ya21F1gB3KqUsiilfg68p7Xe2p1gFouFxYsXA1BeXs6LL74Y82LvcDj4hyUjsCSdBCDVMojnlx5i77ajMT2uEELEUk+nKR4CHG/2ugSYCTwC3ABkKqXGaq1/c+GGSqkHgAcAtNbk5OS0eYCf/OQn/PSnP6W2tpaSkhKmTp3aw8gd+/r9V1PvbmDZ83vIsKVzYF8mX+w5xEMPTcduT+jMzhF2u73dc5ZopmYzNReYm83UXGBuNhNz9bRqWdpYFtRaPwM8c7ENtdbLgGVN25SXl7e77re+9S1efvll/vSnPzF06FAslrYOG313LxnGug8PUnU2hzRrNr//v0cYMKiMWdcWxOX4F5OTk8PFzlkimZrN1FxgbjZTc4G52eKZKy8vr1Pr9XTUTQkwrNnrocDJHu6zFafTyYwZMwDYsGFDtHd/UX+3aBbzbk2lyh+af+dM2SBeefUY1eeiM52yEELEWk8L/SYgXyk1SimVBCwC3u7sxs0fJdiRSZMmhQ64aVNMh1y2xZmcxD8sGcGg3NP4g0Ey7Rn8dbWPj96NzZw8QggRTV0ZXrkcWA8UKKVKlFL3aa19wMPAamAPoLXWnZ5jWGu9Smv9QGfWtdlskf75pUuXxr3YA1x59ThuvjMDdyA0v31dTR6vLz/D1s8Oxj2LEEJ0liXWI1kuRil1M3Cz1vr+kyc71+Pz/PPP43a7sVqtfOc734l5f317/W0Hd5ewcbuddGsyAFV+N1dfaWP42MExzdNRLhOYms3UXGBuNlNzgbnZEtBH32ERTOgUCF1p0Te57777GDx4MIFAgF/96lcxmbu+M8YWDmXJ4sEMGFiGJ+gn0+Zi+5ZkXnvtgDzYRAhhFCPnuunInXfeSXZ2NgAvv/wy7733XsKyzLqugDsWZWNzhj6RpNsG8ME79az7QKZSEEKYIaGFvisXY5uzWCzcfffdzJ8fulH3wIEDvPTSS9GO1yU33VbIdfMdVPtD95RVVQzmzRVn2fS3/QnNJYQQva7rprlx48ZFin11dTXPPPMMFRUX3rwbP2mZLu5eMpox+eeo9tfhtNgoPTmQV147TtmJswnLJYS4tPXKrpvmxo0bxze/+c3I61deeYWXX345gYmgcNpI7l6SR3ZO6Nm0mbZ0Nn5i49XXDnHq6JmEZhNCXHp6faEHSE1N5dFHH2XevNCMyFVVVTzzzDOsXbs2obmumjOem+5Ix287AUCGLZvNGxy8+tohThyJ3Vz7QgjRXK/so2/P+PHjeeCBByJTHO/cuZPf//73MZ8M7WJsNhu33XE5N97mikyWlmHLZuvnSbzy2lGK955IWDYhxKUhoePomwl2dhx9Z5WXl/Paa69FXk+dOpXZs2d3eT/RHhPrrq3jozXFBL15WMP3ANQFGpl8WR3jp4xIWK5oMjWbqbnA3Gym5gJzs8k4+jjKyclp0Xf/xRdf8Mwzz9DQ0JDAVOBKS+WWv7uchXdmkNm/lIagj1SrgwP7Mlm+vIyPV+9JaD4hRN/TZws9nO+7X7RoUWTZsmXLeP755xOYKsRms3H13PHcuSiHlPST1AY8pFmd1JzL5a0Vlbz1h93y0BMhRFT0qT769gwcOJBHHnkk8trtdvPMM8+wd68ZNzXdcFMhixcPoqCwhip/FVaLBasvj3ffrGH58v2Un6xMdEQhRC/WZ/vo21NbW8sLL7zQYtm1117LxIkT25w3JxH9gGdPn2P1h2dItw2ILKvx13PFRB/jJg5LWK7OMjWbqbnA3Gym5gJzs5nYR3/JFfomGzZsYOPGja2WL1iwgPz8/MjrRH4zNTY28tH7B3C7B5NkCX34qva7KRzVwNybZxr5TQ7yA9gdpmYzNReYm00KffviXuib7N69mw8//LDV8tTUVG644QauuOKKhH8z+f1+Nq47yOmygZGROt6gH6xlFOanctnUkQnNdyH5Aew6U7OZmgvMzSaFvn0JK/TNbdu2jXXr1rX53k033cTYsWPjnKgln8/H1s8Os7/EQaatX2R5tb8ep+0cE8e7KJg8PIEJQ+QHsOtMzWZqLjA3mxT69hlR6Js0NDSwZcsWtmzZ0ub7N954I+PGjYvbs2vb0lgf4KMP91Jek0qmLSuyvMpfS39XFddcN4pklxObzRb3bPID2HWmZjM1F5ibTQr9Bbrz4JF469+/P5988kmb3TsWi4Xrr7+e8ePHx72gNv9mOnrgFHv2VnKuNgeXNSmyjjvgJdN1huvn5eNMTmpvVzHNZhJTc4G52UzNBeZmk0LfPqNa9M01/08LBoPs37+f1atXt7numDFjmDNnDsnJyXHN1dzhvSfY/mUVft8gnJbQLx9fMEBdoJ7CUQ0Mzksnd8TAhGRLNFNzgbnZTM0F5mYzsdDbYx+l77BYLBQUFFBQUEAwGOTw4cOsWbMm8vzaQ4cOcejQIQCmT5/OlClTcLlccc04evwQRo8fgs/no6ykgrWfubFaUsiwuSg55qLkGFR9eox0Zy2zZg1k0JCcuOYTQsSftOg70Nnfzm63mzVr1nD8+PFW702cOJEZM2ZEJluLZ64mf/twDzW1QfyevBbLq/w1zJxkIW9kf1xpqQnJFi+m5gJzs5maC8zNJi36PszlcnH77bcDoQnVtm3bxu7du4HQLJo7d+4EICUlhfnz5zN48GAcDkfc8s2+4TIAPA1ePPUe3n3/JAEyybSls3cX7N3lpcp/huED6kl2WpleNCYhF3KFENEnLfoO9PS3c3V1NZ9//jl79rQ9Wdno0aO5/PLLGTVqVFxzNVnzzm7cDWD3X9jSr8VhrWX4IAtfua4gIdmizdRcYG42U3OBudmkRX+BZqNuEhkjpjIyMpg7dy5z584lGAyye/dudu/ezalTpwA4fPgwhw8fBsDpdFJQUEBhYSEDB8b2gmmTeQsLgdC0CyePVVJT7aXs9ADSrC5sljTKT8Mrr5VA0MdtCweQlhnfaw5CiJ6TFn0HYvnbub6+nhMnTrBmzRp8Pl+r910uF5MnT2b48OFkZ2e36EqJR6vhr2v2cLw8jRSrKzIFgzcYwBts5NqrrFjtVvoPzGjVBSUtra4zNZupucDcbNKiFy2kpKQwduxYxo4dSzAYpLa2lsOHD7N582bcbjdut5vPPvuMzz77DAhGYy6HAAASkElEQVSN6Z89ezaDBw+OS75r5oX69RvcHla9cxh/wIqV/qRZnWzeEFqnJnCWy0fXAZCR6WTU+CFxySaE6Dxp0XcgUa2GYDBIRUUFR48eZefOnVRVVbVaJz8/n8zMTHJzc8nLyyMpKSnmd+vWVrn528dH8AcseDznJ1trYks+wYD+qXg8XiZdMYSMrOiNNOopU1uAYG42U3OBudmkRS86zWKxkJ2dTXZ2NtOmTcPn83H48GG2b99OZWUlDQ0NHDhwoNV2I0eOJCsri7y8PAYOHEhGRkZUc6Vlulhw6+VAqF9/947QtYbSs5BMLv6GIZSGf2e/824NA7OOAZCcZOUr1+XLSB4hEkBa9B0wtdWQlpbGiRMnKC0tZd++fVRXV1NRUdHmumPGjMHpdDJx4kR8Ph+DBw+OesH1+/0c2nWC+nofrtRUduxKwnFBa98dKCM7I/TUrKF5Li6b2vln5EaDqf+XYG42U3OBudmkRS+iJjk5mczMTDIzMykoCA1/DAQCVFdXc/r0aY4fP86ZM2c4ffp05G7dpnH9AElJSYwePZrBgwdTWFiIxWLpUfG32WyMmxSaOTMnJ4cBecWcPhXqbjpd5qaifBAu6yAaakPr79jbSGnZ+TzpaQ6u+Gp+q/0KIXpOCn0fYrVaycrKIisri3HjxgHg8Xiw2Wzs2rULm83G+vXryczMpLS0lL1797J3714+/vhjALKyshg5ciSNjY3MmjWLhoYGXC5Xt+buycxOJzM7HYD8CVB6rJza2tCD2bfsqCXVOpjaqvNj92ur4OPVe0hPC43gsdksFE4dTpIzfjeVCdFXSaHv45xOJwCTJ08GYMKECQBUVlbi9XrZuHEjycnJ7Nmzh3PnzrFt2zYAdu3aFdnH+PHjCQQCXH311dTV1WG328nKyqIrBg8/P6fO8DGNHN5zkqZuw+Mn6vC486g5l0vNufPbHCnZz8wZ5x+nmJGdFrVpGoS4lEihv0T16xd6cMnNN98MwNVXX43P5yMpKYlNmzaRlJTEtm3bqKurizxEff/+/ZHtJ06ciNvtZtSoUYwbN46amhogNAS0I0lOB+OnnO+fHzfJz46NR/A0nL+X4FTZAOz+IZFhnAC1gRoWL5ZCL0RXyXz0HbiUL/gEg0GCwSCNjY1s3boVp9PJyZMnOXz4MA6HIzJrZ3NTp07F4XBQUVHBtGnTGDRoEF6vF5/P16WZPDf9bT9nzp7ff0VdKpm2fjQEW95Y5gk0sEgNxm7vuM1i6v8lmJvN1FxgbjYTL8bKqJsOyDdTaz6fD7vdTnFxMUeOHCEtLY2Kigr27dsHhC4UNzSE+uPtdnvkrt9rrrkGm82G2+0mJyeHMWPGdPqYR/ed5LOttQSD57+nA6SQaUujyl+JhebfxwGKpqcysqDl/D2m/l+CudlMzQXmZjOx0EvXjeiyptbzqFGjIpOxBYNBrrrqKpxOJ3l5eaxcuZLq6mrS0tKoqanh0KFD/PWvf22xn7Fjx+J2u6mpqcFut7No0SJ8Ph91dXV4PB4GDRoUOdaIgjxGXDC32q4txWzf34DNcr47x4IFlzWJz7aUc6S45URySclJjC/Mof/AzGifEiGMJoVeRIXFYiE9PT3y+qtf/Wrk38FgkJKSEux2Oy6Xi3379rFx40bOnj2Ly+XC7XYTDAb5zW9+02KfhYWFjBgxgrq6OtxuN1lZWRQWFkbev3z6KC6f3jKH3+/nj2+cI92WQ82FNxNXwZpTpSxaLIVeXFqk0IuYs1gsDBs2LPJ6xowZzJgxI/La7XazadMmHA5HZDjnmjVrIjN9NldVVUVdXV3kT2pqKgsXLoxM/WC1Wim6ys+58ppWOXbsSyXZMpA3VrT+WN0Y9LFwQZpRUzYIES1S6EXCuVwurr322hbLMjMz8Xq9uFwuUlNT2bNnD59++imbN28mJSWF1NRUvF4vZWVlrFy5Eo/HQ319PXV1dfj9fhYuXMjo0aMJBoOR946dOMjZ6jQslpbf9gGcZNoyWLP6OCnOQJsZxxdkMuayobE6BULElBR6YaTc3NwWr6dNm0ZhYSFOpxOrNTS1wtGjR1m7di319fW4XC6ys7NxOBzs2LGDd955B5fLRX19PYHA+eI9ffp0rrrqqkjxr6+v58Cuo5w6WY0reSgBT9t51m89x5jLYvblChFTUuhFr2CxWEhJSWmxbMSIEXz9619vsaypBe/1eklJSYm0/lNSUli7di1btmxh69attDXabOHMW3E4HHi8nvA+PHi8Hr7cV0e6cxivvtb+yLC8flVct0B+EwgzSaEXfYrFYuHGG29s8z2Hw8GRI0civwCa/pSVlbFhwwbeWfNWu/ttbDxNWvqVEPQRCHoJBrwEgo0EA14cWDl+Jp3y0nPtbg+Qme2K63OChWgihV5cMmbOnNnm2P0BAwZQX1+P3W4nOTmZlJSUFn+/9957lJeXUlX5TotuoObSUwpY/9dQH37oRrNG/EEvgYCHQMCDP+jFi417750Y069RiLZIoReXvNTUVK655pp2358xYwbHjh1rUfyb/1mx4nVq6vfR0Hgcf6CRQMAHtO4actiz2fZ521M4pDhPUlV7Dp+vkYLLhpM9qF+0vjwhpNAL0ZH8/Hzy89ufQnnKlMmUlZVFCr/T6Wz1y+Cttz6isfEsW7fvJRD0Egh4CQS9+AOeUFdQs6kdtu/N5aEH74zHlyYuEVEv9Eqp0cDjQKbW+o5o718I01x11VUdrlM4fgTbd1bi8ZVgszmwO+zYrA5sthTsNgdOZzLBoIWTZ0rwNVbw+d+2t/WhoBVXWjITphV0vKK4pHWq0CulXgAWAqe11hOaLZ8PPA3YgN9qrZ/SWh8G7lNKvRmLwEL0RtdcdzXXXHd1u+83zY+ydOlyGv21fP7FX9td90K1NfXMumZKNGKKPqqzLfqXgGeB3zUtUErZgF8Dc4ESYJNS6m2t9e429yCE6ND8G4rYvu0QFix0NFfVObebquqDlFdUxyec6LU6Vei11uuUUiMvWHwlcDDcgkcptQK4FZBCL0Q3jcofzqj84Z1ad9vGPazbcJDDx7fzzDM7On2M1JRhfPP+W7sbUfRCPemjHwIcb/a6BJiplMoGngSmKqUe01r/rK2NlVIPAA8AaK3Jyclpa7WEs9vtRmYzNReYm83UXNC9bNfO+wpf7jmK19f6uQDtqas/jbexttPH6mvnLB5MzNWTQt/W58qg1vos8GBHG2utlwHLmrYzcV5pkDmvu8PUbKbmgu5n+4d753dp/aVLX8fvr+I/fv5ip9a3WCwt7iLun57OberaLh0zVkz9/0zAfPQd6kmhLwGGNXs9FOjS00OaPWGqBzGEEJ2V7HRRU3uaWveRbmwdxF2XBFwb3VAi5npS6DcB+UqpUcAJYBGwpCs70FqvAlYB9/cghxCik77xvxZ2af3mrdPnf7sKd92RGKQSsdbZ4ZXLCf0az1FKlQD/rrV+Xin1MLCa0PDKF7TWu2KWVAiRUDarFQjy3HOv9Gg/FmsS995zM6mpKR2vLKKis6NuFrez/F3g3e4eXLpuhOg9CsYMYdvuGoKduZOrHcGAB5+3giP7Siic2v7dxiK6EjoFgnTdCNF7fOWaKXylhzdm/fH1tZSU7aShoZ2J/0VMyFw3Qoi4cThsAGzYtpON2/d0sHYXWSx8dcZEmRKiDQkt9NJ1I8SlZdy4IRwrPUow4MFPNFv1Qfz+GrbtTJNC3wbpuhFCxE3BhDEUTGj9TIDuaD4iqKqimpdfeQl/O88LuNRJ140QotdLdjkB8Pl9lBSfiumxrHYrecMGxfQY0SaFXgjR6zmdTizYcNcd4Y+rjsT8eDn9Cljyj20/stJE0kcvhOgT8kdO4ey52pgf5+y5A9R5eteoIemjF0L0CfNvKYrLcX71q8MEg73rWoA10QGEEKI3sWBrMdFbbyB99EII0RUWKx5PKb/+9UuJTsKTT/5rp9aTPnohhOiCfhlDqa6rafd9C5163G9cSR+9EEJ0wd33XPwZACbOky999EII0cdJoRdCiD5OCr0QQvRxcjFWCCH6OLkYK4QQfZx03QghRB8nhV4IIfo4KfRCCNHHWQyZs8GIEEII0QtZOlrBiBa9UmoLobDG/TE1m6m5TM5mai6Ts5may+RsCcjVISMKvRBCiNiRQi+EEH2cKYV+WaIDXISp2UzNBeZmMzUXmJvN1FxgbjbjcplyMVYIIUSMmNKiF0IIESMJf8KUUmo+8DRgA36rtX4qxscbBvwOGAwEgGVa66eVUv2B14GRwBFAaa0rlVKWcL6bgDrg61rrreF93Qv8W3jXT2itX45CPhuwGTihtV6olBoFrAD6A1uBf9Rae5VSzvDXMR04C9yltT4S3sdjwH2AH3hUa706CrmygN8CEwgNh/1fwD4SfM6UUt8FvhnOtBP4BpBLAs6ZUuoFYCFwWms9Ibwsat9XSqnpwEtACvAu8E9a6059JG8n2y+AmwEvcAj4htb63MXOR3s/r+19n3YnV7P3vg/8AhigtS434ZyFlz8CPAz4gD9rrX8Yz3PWHQlt0YeL2q+BBUAhsFgpVRjjw/qA/621vgyYBXwnfMwfAR9prfOBj8KvCWfLD/95AFgazt4f+HdgJnAl8O9KqX5RyPdPwJ5mr38O/Hc4VyWhbyTCf1dqrccC/x1ej/DXsgi4HJgPPBc+zz31NPC+1no8MDmcMaHnTCk1BHgUuCL8g2gj9LUn6py9FN6+uWieo6XhdZu2u/gTMDrO9gEwQWs9CdgPPBbO0Ob56ODntb1z3p1cTQ2yucCxZosTfs6UUtcBtwKTtNaXA78ML4/nOeuyRHfdXAkc1FofDv8mW0HoJMaM1vpUUytAa11DqGANCR+3qXX5MnBb+N+3Ar/TWge11huALKVULnAj8IHWukJrXUnoh6Yr30StKKWGAl8j1HIm3IK5HniznVxNed8E5oTXvxVYobX2aK2LgYOEznNPcmUAVwPPA2itveGWX8LPGaFPpSlKKTuQCpwiQedMa70OqLhgcVTOUfi9DK31+nCL9HfN9tWtbFrrNVprX/jlBmBos2xtnY82f147+D7tcq6w/wZ+SMubKRN+zoCHgKe01p7wOqebZYvLOeuORBf6IcDxZq9LwsviQik1EpgKfA4M0lqfgtAvA2BgBxljkf3/J/TNHQi/zgbONfthbH6MyPHD71eF149FrtHAGeBFpdQXSqnfKqVcJPicaa1PEGpRHSNU4KuALZhxzppE6xwNCf87Fhkh1BX3XjezXez7tMuUUrcQ6rrcfsFbJpyzccBspdTnSqm/KqVmdDNbVM9ZRxJd6Nu6qysuw4CUUmnAH4B/1lpXX2TV9jJGNbtSqqkvcEsnjh23XGF2YBqwVGs9FXBzvguiLfE6Z/0ItaRGAXmAi9BH5PaOEc9z1pGuZolZRqXU44S6NF9NdDalVCrwOPCTNt424ZzZgX6Eun1/AOhw69yEbO1KdKEvAYY1ez0UOBnrgyqlHISK/Kta6z+GF5eFP+oR/rvpI1l7GaOdvQi4RSl1hNDHu+sJtfCzwt0SFx4jcvzw+5mEPmbG4pyWACVa68/Dr98kVPgTfc5uAIq11me01o3AH4GrMOOcNYnWOSrhfNdK1DKGL2IuBO5udpGyq9nKaf+cd9UYQr+4t4d/FoYCW5VSg7uRKxbnrAT4Y7j7aCOhT9853cgWzXPWoUQX+k1AvlJqlFIqidDFjLdjecDwb9/ngT1a6/9q9tbbwL3hf98LvNVs+T1KKYtSahZQFf4IvhqYp5TqF25Zzgsv6xat9WNa66Fa65GEzsNftNZ3A2uBO9rJ1ZT3jvD6wfDyRUopZ/iqfj6wsbu5wtlKgeNKqYLwojnAbhJ8zgh12cxSSqWG/1+bciX8nDUTlXMUfq9GKTUr/LXe02xf3RIeDfIvwC1a67oLMrd1Ptr8eQ2fw/bOeZdorXdqrQdqrUeGfxZKgGnh78GEnzNgJaFGGEqpcUASoaKdsHPWGQkt9OH+qYcJ/UftCS3Su2J82CLgH4HrlVLbwn9uAp4C5iqlDhC62t80zPNd4DChiyv/A3w7nL0C+D+E/iM3AT8NL4u2fwG+p5Q6SKhf7/nw8ueB7PDy7xHuSgmfP02o4L0PfEdr7Y9CjkeAV5VSO4ApwP9Hgs9Z+BPGm4SGpu0k9P28jASdM6XUcmA9UKCUKlFK3Ud0z9FDhC7UHyQ0HLKpT7272Z4F0oEPwj8Hv7nY+ejg57W9c96dXO0x4Zy9AIxWSn1J6JP3veHWfdzOWXfInbFCCNHHJbrrRgghRIxJoRdCiD5OCr0QQvRxUuiFEKKPk0IvhBB9nBR6IYTo46TQCyFEHyeFXggh+rj/B7+We5xjkbXtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = s['count'].to_frame('freq')\n",
    "factors = [1e-2, 1e-3, 1e-4, 1e-5]\n",
    "for f in factors:\n",
    "    sf = make_sampling_table(vocab_size, sampling_factor=f)\n",
    "    df[f] = df.freq.mul(sf)\n",
    "df[factors].plot(logy=True);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:58:07.030148Z",
     "start_time": "2018-12-10T06:58:07.026839Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sampling_table = make_sampling_table(vocab_size, sampling_factor=SAMPLING_FACTOR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T06:58:07.362609Z",
     "start_time": "2018-12-10T06:58:07.192858Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucHFWd9/HPyQyXXAmhuU0SSJCgBLwgiLDeUNENyhJ3V4+AF1Tc7PqIwqKr8siDPOCyqOuy7Aq4EdmIi2Z/oiLPYxRR8YZooghouCUkQzIzCclMMrlDmOTsH6cGOp3u6Z5J91RV9/f9evVruqrPVP3qdHf96pw6XeVCCIiIiGTNmLQDEBERKUcJSkREMkkJSkREMkkJSkREMkkJSkREMkkJSkREMkkJKueccwuccz8e4vUZzrngnHv1aMYltXHOdTrnLi+a/plz7uY0Y6qHap/LYSxnj/qpUGaPOitdt3PuSufc8hrW1RR130yUoDLMOTfWOXe1c26Zc26Hc67PObfEOffRYSxmNXAk8Ns6xTTdOfcl59xy59zTzrn1zrnFzrlPOucK9VhHvdSp/kbbXwGXjsaKkh1ySB47nXNPOOf+yTk3bjTWX0fV6uyfgdMGJ5xzlzvnOkewHBll7WkHIEO6CXg9cDHwIDAJOAk4qtYFhBB2AWvrEYxz7mXAT4AngU8BS4kHOS8AzgU+AHy+wv/uH0LYWY84hmGf62+0hRA2jPIqvwF8DNgfeB0wn1hPHy5XOKX3cUjV6iyEsBXYuq/LkRSEEPTI6APoBy6qUmYB8OOi6aOBRwADDgBmAAF4dfL64PR7iMlmB7ASeFeV9TjgIeKOvr1SmaLnncBngRuBPmBJMv9i4AHiDmMtsBA4suj/zkjiewtwXxLf74ETksevgO3AYmB2Herv5cAPgHVJTEuAOSVlOoGriQlvU1L2oqR+/x3YCHSXrivZjouBbwPbgB7g0jLLvrxo+mfAzaXTwP9J6mtD8p6PLyozBrgGWJ9sw0LgEmCgyrbvsa5k3leANSXvxVuTen96cBuT9+f3wDNJfdxYEtMC4MfEFkl38p59GyiMoO7/MamDzUAv8DlgzBB1toA9vxNXAsuT5+9Ltqn4ceUQ9fER4NFk25cBn6bo8w/MBf6QbF8/8XN5Utr7jmZ5qIsv29YAc5xzU2op7Jx7KXGnfjdwbgjhmSGKfw64BXgZcBvwdefcKUOUfynwYuBzIYSBcgVC8o0t8lHizud04IKi+R9PlvWXxNbMwjKL+0fizuBkYCfwTWKC+EzRvP8cIl6orf4mJes/g7jDvAu40zl3XEm5jxB3UCcD/5Y8vktM7q8AvgT8m3Nudsn/fYa44zuJWOefd879VZW4S70dmJLEeD7wNuATRa9fQqzrS5P1LAauGOY6Bu0A9iuZ90Viy/h44A7n3EuAO4FfED8/FwBnA18u+b9Tk5jnEBPaS4ifuUHDqfseYj3/PfHg4JIRbt9/E9+HLmLX95HELsC9OOeuJH5WLyNu+8XA3xLfU5xzRwDfIn42TyB+zv8VKPv9kBFIO0PqUfkBvIrYnbaL2HqZTzxiK26pLCAeqb6eeAR3WckyZlC+BXV1SblfA/81RCw++b+TSuZ3EY9+twI/KJrfCfykhm08KVnu1GT6jGT6bUVl3pHM++uieX+ZzJuwL/VX4f8eBD5dsi13FE2PIR7N/7+SeRspakUl8X29ZNnfAH5VsuxqLaiHSpbxZeC+ounuMu/nQobRgiK2kE8nttAWlrwX7yn5v68Di0vmzQV2A0cXfS63AgcVlXlzsrxZw6z7X5aUuQboGqLOFlChBZVMXw50VqmPccRWUWmL7r1Af8lnd0at32k9hvdQCyrDQgj3Es/vvAb4GnA4sZvkTuecKyr6YuCHxB3dP9W4+PtKpu8FZgM4577snNta9DiKuAMr5zXEo+hFwNiS1xaXFnbOneGcu8s5t9o5t4XYdQSxa7LYg0XPB8+hPVRm3mEV4qqp/pxzhzrnbnTOPeqc63fObSUeDVeMJ4Swm9id9lDJvHVl4qlYz8PwQMl0d7ItOOcmAR3Ab6qst5ILkm1+Gvglsdv3opIype/jCcTWU7GfEz8jxdv2cAhhU9H0vcnf45PYa637cnU4Ndn2RjmB+Hn+dvF3AfgP4CDn3KHE9/8u4E/Oue865y52zk1vYEwtRwkq40IIAyGEX4cQvhhCmEvsQz8beG1RsVXEndi7nXMHjXBVxQnoCmLSGXz0AI8lr+2xcw0hrAwhLCe2KEpt22MFMdEtIh4VnwucApyTvLx/yf8+W7yaIeYN+Rmuof4WEBPYJ3g+2T5QJZ7B9ZebV+07VSnRD6V0UELxelzRvJH4LnGbZwEHhhDeEULoLSmzbe9/q7i+4cSxgNrqvtRI6nC4Buv3Hez5XXgxsa42hDgA6SzgDcTzZ38NPO6cO3sU4msJSlD580jyt/hIfRPwJmJX1o+dcwfXsJzTSqZPH1x2CGFdCGF50WOA2IL4E/Ap51zpOYpavYJ4VHpJCOHeEMJjJC2BUVRaf68Fbgwh3BlC+CPxvNUxdVxfxXquh6SF0pMsd6j1VrI5eY9XhQrnFstYShzxV+x1xOT0cNG840taOX+W/B3c/lrrvlwd9oQQyh0U1WIn0FalzFJiq/KYku/C4GMXxPOuIYTFIYRrQgivJbYk3z/CuKSEhplnmHPu58QTsL8jdikdS+x/7wfuKS4bQtjsnPtz4PvAT51zZ4YQ+oZY/IXOuUeTZb+b+KWveOI5hBCccxcQu4CWOOc+S/wSDxBPfr+a2PU0lGXEndjHnHO3EQdejPRkflU11t9jwLucc78i7rSuovrOazjOds5dROwKmgO8k9h6rKcvAv83eT8XE0fdDZ7vaYQvAPc75/6FeF5vBnE0420hhFVF5QJwa/JD2ynADcD3QwjLktdrrfuXJQMWvkFsdV9MPK80UiuBI5xzpxM/k9tDCNuLC4QQtjrnrgGuSXqD7ybuL19MPA/7SefcnwFvBH5ETK6ziN+Fr+5DbFJELahs+wHwLmK32GPEUWvLgFeV6YYhxN97nEUc1n2Pc67i+Rni75jmEfvR3wtcEEJYMlQwIYT7iUnl18SRUA8kj8uJo5nOq/L/DxFHZP0t8Uj744x8NFYtaqm/9xO/B4uBO4jn8oash2G6CjiT2AL938RBLLfXcfkQR459CbieOOT5NGLSerrO6wGeex/PIbaaHiQOmvg+8HclRRcTzzHeTUzQS9mzdVFr3f878bzU74jbeRNw3T5swh3Ez+v3iQcunyhXKIRwNXHU4AeJ2/mrZLozKbKJeGD3PeLn6hbiiNir9yE2KeJCaNRBlmSRc24G8QjyNSGEXw1dWvaFc25wBNx/pbDuW4CXhhBOHu11i9SLuvhEcs4510Ecdn8P8TzkXxBbxaWj8URyRQlKJP92EUebXQ0cCCwHPhRC+EqqUYnsI3XxiYhIJmmQhIiIZFKaXXxquomItK6qP7hO9RxUT09PmqsflkKhQG/vXiO7Mytv8UL+Yla8jaV4Gy+tmDs6Omoqpy4+ERHJJCUoERHJJCUoERHJJCUoERHJJCUoERHJJCUoERHJpKrDzL33txBv8LbOzE4s87ojXkX5LcRbJL/PzO6vd6AiItJaamlBLSDex6aSs4j3QZlFvH3DTfseloiItLqqLSgz+4X3fsYQReYCt5pZAH7jvZ/svT/SzNbUK8h9FXbtYvf8z8PGoe7fN7S+9nZ2DdR6w9H05S1eyF/MirexFG/jpRbzl75RU7F6XEliKrC6aLormbdXgvLezyO2sjAzCoVCHVZf3bOPL2XD/ffRPms2YyZMqv4PZTjnGJOjC+vmLV7IX8yKt7EUb+NlPeZ6JKhy11Mqu8VmNp94i2iAMFqX2Ni95N749+8+SZh08IiWkbfLmOQtXshfzIq3sRRv42U95nqM4usCphdNTwMydZG98PhSOGIqboTJSURERl89WlB3Ahd57xcCrwQ2Zer80+5dsOxh3CtenXYoIiIyDLUMM/8mcAZQ8N53AZ8B9gMwsy8Di4hDzJcTh5m/v1HBjkjXk7BjG8w6Ie1IRERkGGoZxXdeldcD8OG6RVRnYdlSANxxSlAiInnS9FeSCI8vhUMOw005NO1QRERkGJo6QYUQYNlSnLr3RERyp6kTFOvXwJZNcOzxaUciIiLD1NQJKjzxGADuBS9MORIRERmupk5QrHgUDhwLHUelHYmIiAxTUyeo8MSjMPM43Ji2tEMREZFhatoEFZ7eAV1P4l7worRDERGREWjaBEXnMgi7cccoQYmI5FHTJqjwxKPxyTEaICEikkfNm6BWPAZHTMONn5B2KCIiMgJNmaBCCLDiMQ0vFxHJsaZMUKxbA1s3g84/iYjkVlMmqMHzTxrBJyKSX02ZoFjxKIwdB0dOr15WREQyqSkTVHjiseQHuk25eSIiLaHp9uDh6e3Q/aR+/yQiknNNl6BYmfxAVyP4RERyrekSVFgRr2DOTCUoEZE8a74E1bkMjpiqH+iKiORc0yUoVi7DzZiVdhQiIrKPmipBhY19sGkDzDgu7VBERGQfNVWCYuXjALgZx6Ybh4iI7LOmSlChcxm0tcFRx6QdioiI7KPmS1BTZ+D22z/tUEREZB81TYIKu3dD53I0QEJEpDk0TYJi3RrYsQ1mKkGJiDSDpklQoXNwgIQSlIhIM2iaBEXncjjgQOjQFcxFRJpB0ySo0LkMjjoGN6Yt7VBERKQOmiJBhYEBWLUCde+JiDSPpkhQ9DwJz+6EmbqChIhIs2iKBBVWLgM0QEJEpJk0RYKicxlMmAiFw9OORERE6qS9lkLe+znA9UAbcLOZXVvy+lHA14DJSZlPmdmiOsdaUehcDkcfi3NutFYpIiINVrUF5b1vA24AzgJmA+d572eXFLscMDM7CTgXuLHegVYSnt0Ja1bhjtYFYkVEmkktXXynAsvNbIWZ7QQWAnNLygRgUvL8IKCnfiFW0f0k7NqFO+oFo7ZKERFpvFq6+KYCq4umu4BXlpS5EviR9/4jwHjgzLpEV4Ow6on4RFcwFxFpKrUkqHIndkLJ9HnAAjP7ovf+dODr3vsTzWx3cSHv/TxgHoCZUSgURhLzHjavW8PT4yZQeNEJDT0H1d7eXpd4R0ve4oX8xax4G0vxNl7WY64lQXUBxdcPmsbeXXgXAnMAzOw+7/2BQAFYV1zIzOYD85PJ0NvbO5KY97Dr8aUwfSZ9fX37vKyhFAoF6hHvaMlbvJC/mBVvYynexksr5o6OjprK1ZKglgCzvPczgW7iIIjzS8qsAt4ILPDeHw8cCKyvOdoRCgMDsHol7g1vbfSqRERklFUdJGFmA8BFwF3AI3GWLfXeX+W9Pycp9jHgb7z3DwLfBN5nZqXdgPW3tgsGnoXpOv8kItJsavodVPKbpkUl864oev4w8Kr6hlbd4AAJd7RG8ImINJt8X0li1QrY/wA4vLb+TBERyY9cJ6iw6gmYPlO32BARaUK5TVBh925YtRKn3z+JiDSl3CYo1q2BZ3aAriAhItKUcpugnhsgoQQlItKUcpugWLUC2tqhY3r1siIikju5TVBh1RMw9Whc+35phyIiIg2QywQVQoDVK3DTZ6YdioiINEguExSbNsLWLaAEJSLStPKZoLo6AXDTZqQahoiINE4uE1To7oxPph6dahwiItI4uUxQdHXC5ENwEyZVLSoiIvmUywQVujpB3XsiIk0tdwkqDDwLa7p0/klEpMnlLkGxtht2DagFJSLS5HKXoIJG8ImItITcJSi6OuMljg6fmnYkIiLSQLlLUKG7E46cjmuv6WbAIiKSU7lLUHR1qntPRKQF5CpBhS2boX+DBkiIiLSAXCUokitIqAUlItL8cpWgBkfwqQUlItL8cpWg6H4SJkyCSZPTjkRERBosVwkq9KyKNyl0Lu1QRESkwXKToEIIsGY17kjd4l1EpBXkJkHRvwF2bIcOJSgRkVaQnwTVswoA13FUyoGIiMhoyE2CCmtigkJdfCIiLSE3CYqe1TBhIkw8KO1IRERkFOQmQYWeVfEafBrBJyLSEnKRoJ4bwafzTyIiLSMXCYpNG2H7NjhSCUpEpFXkI0E9N4JPAyRERFpFLhJUWLM6PlEXn4hIy6jprn/e+znA9UAbcLOZXVumjAeuBALwoJmdX7coe1bDuAm6Bp+ISAup2oLy3rcBNwBnAbOB87z3s0vKzAIuA15lZicAl9QzyLBmFXRoBJ+ISCuppYvvVGC5ma0ws53AQmBuSZm/AW4ws40AZrauXgGGEKBHI/hERFpNLV18U4HVRdNdwCtLyhwH4L2/l9gNeKWZ/bB0Qd77ecA8ADOjUChUXfmu/g30btvChGNfxLgayjdKe3t7TfFmRd7ihfzFrHgbS/E2XtZjriVBletXC2WWMws4A5gG/NJ7f6KZ9RcXMrP5wPzBZfT29lZdeXjsTwBsm3gw22so3yiFQoFa4s2KvMUL+YtZ8TaW4m28tGLu6OioqVwtXXxdQPH47mlAT5ky3zOzZ81sJfAYMWHts/BUd3xyxNR6LE5ERHKilhbUEmCW934m0A2cC5SO0LsDOA9Y4L0vELv8VtQlwqe6oX0/mJLdZqiIiNRf1RaUmQ0AFwF3AY/EWbbUe3+V9/6cpNhdQJ/3/mHgHuAfzKyvHgGGp3rg8A7cmLZ6LE5ERHKipt9BmdkiYFHJvCuKngfg0uRRX2u7YerRdV+siIhkW6avJBEGBqB3Le7w2k6oiYhI88h0gqL3Kdi1SwMkRERaULYTVDKCzx2uBCUi0moynaA0xFxEpHVlOkGxthsmTMKNn5h2JCIiMsoynaAGh5iLiEjryXSC4qlunLr3RERaUmYTVNixPd7qXQMkRERaUmYTlEbwiYi0tswmqPBUcj1adfGJiLSkzCYonuoB5+DQI9KOREREUpDdBLV+LUw+BLff/mlHIiIiKchsggq9a9V6EhFpYZlNUKxfi1OCEhFpWZlMUOGZp+MQcyUoEZGWlckExfq18a8SlIhIy8pmguqNCcodemTKgYiISFoymaDCuqQFdZhaUCIirSqTCYr1a2HceF3FXESkhWUyQYX1a6Cg1pOISCvLZIJi/VMaYi4i0uIyl6DC7l3Qt07nn0REWlzmEhQbemHXAGgEn4hIS8tegkp+A+UKh6cciIiIpClzCSoM/kj3MLWgRERaWeYSFL1PQVsbHHxI2pGIiEiKspeg+tbH22yMaUs7EhERSVHmElTYsA4OOSztMEREJGWZS1D0rccdcmjaUYiISMoylaDCwAD0b1ALSkREspWg6O+DsBumqAUlItLqspWg+tYD4NSCEhFpeZlKUKFvXXyiBCUi0vIylaDYkCSoKYV04xARkdS111LIez8HuB5oA242s2srlHs78C3gFWb2u2FH07ceJk3G7bf/sP9VRESaS9UWlPe+DbgBOAuYDZznvZ9dptxE4KPAb0caTOjTb6BERCSqpYvvVGC5ma0ws53AQmBumXJXA58Hnh5xNH3rcRrBJyIi1NbFNxVYXTTdBbyyuID3/iRgupn9f+/9xystyHs/D5gHYGYUCs+fawohsG5jL2NPP4OJheydg2pvb98j3qzLW7yQv5gVb2Mp3sbLesy1JChXZl4YfOK9HwNcB7yv2oLMbD4wf3AZvb29zy9w80Z4dic7xk3gmaL5WVEoFOjNYFyV5C1eyF/MirexFG/jpRVzR0dHTeVq6eLrAqYXTU8DeoqmJwInAj/z3ncCpwF3eu9PqSmCQfoNlIiIFKmlBbUEmOW9nwl0A+cC5w++aGabgOfaiN77nwEfH/Yovo1JFj84u81NEREZPVVbUGY2AFwE3AU8EmfZUu/9Vd77c+oVSNi4IT7RfaBERIQafwdlZouARSXzrqhQ9owRRdLfB+3tMGHSiP5dRESaS3auJNHfBwdNwblyYzJERKTVZCZBhY196t4TEZHnZCZB0b8BN1kJSkREokwkqBBC7OJTghIRkUQmEhQ7tsMzT8PBU9KOREREMiIbCaq/L/5VC0pERBLZSFAbY4LSOSgRERmUiQQVBltQGsUnIiKJTCSowRYUk3UOSkREomwkqP4+mDBRd9IVEZHnZCJBhf4NGiAhIiJ7yESCon+DuvdERGQP2UhQW/pxEyenHYWIiGRI6gkqhACb+2GSEpSIiDwv9QTF9m0wMKAEJSIie0g/QW3uj3+VoEREpEhmEpRTghIRkSKpJ6igFpSIiJSReoJSF5+IiJSTjQTlxsCEiWlHIiIiGZJ+gtrSDxMn4ca0pR2JiIhkSOoJKug3UCIiUkbqCUo/0hURkXIykaA0xFxEREqlmqB0mSMREakk3RbUMzvg2Z1KUCIispd0E9SWzfHvhEmphiEiItmTboLatgUAN16/gRIRkT2lm6C2xgSlH+mKiEipdAdJJC0oxquLT0RE9qQWlIiIZFLK56CSQRLjJqQahoiIZE/6Lahx43Ftug6fiIjsqb2WQt77OcD1QBtws5ldW/L6pcAHgQFgPfABM3uy6oK3bQGN4BMRkTKqtqC8923ADcBZwGzgPO/97JJifwBOMbOXALcDn69l5WHrFv0GSkREyqqlBXUqsNzMVgB47xcCc4GHBwuY2T1F5X8DvLumtW/bAhMPqjlYERFpHbWcg5oKrC6a7krmVXIh8IOa1r51M04j+EREpIxaWlCuzLxQrqD3/t3AKcDrKrw+D5gHYGa47VsZWziMiYVCjeGmp729nUIO4hyUt3ghfzEr3sZSvI2X9ZhrSVBdwPSi6WlAT2kh7/2ZwKeB15nZM+UWZGbzgfnJZAg7trOjrZ1nenuHF3UKCoUCvTmIc1De4oX8xax4G0vxNl5aMXd0dNRUrpYEtQSY5b2fCXQD5wLnFxfw3p8E/Acwx8zWDStSjeITEZEyqp6DMrMB4CLgLuCROMuWeu+v8t6fkxT7AjAB+Jb3/gHv/Z01RzB2/PCjFhGRplfT76DMbBGwqGTeFUXPzxxpAE4JSkREykj/lu9jx6UdgYiIZFD6CWqcEpSIiOwt/QSlLj4RESkj/QR1oFpQIiKyt/QT1NixaUcgIiIZlG6COmAsboxutSEiIntLN0FpBJ+IiFSgBCUiIpmUboIapxF8IiJSnlpQIiKSSakmKKch5iIiUkG6LagDNcRcRETKSzdB7X9AqqsXEZHsSvl3UEpQIiJSnlpQIiKSSSknqANTXb2IiGSXWlAiIpJJKZ+DUgtKRETKS/d3UGpBiYhIBRrFJyIimaRzUCIikkk6ByUiIpmkFpSIiGRSuglqv/1TXb2IiGSXEpSIiGRSugmqvT3V1YuISHalm6DalKBERKQ8taBERCST1IISEZFMSvdSR86luXoREcmwdFtQIiIiFShBiYhIJilBiYhIJilBiYhIJtU0jM57Pwe4HmgDbjaza0tePwC4FTgZ6APeaWad9Q1VRERaSdUWlPe+DbgBOAuYDZznvZ9dUuxCYKOZHQtcB3yu3oGKiEhrqaWL71RguZmtMLOdwEJgbkmZucDXkue3A2/03msMuYiIjFgtXXxTgdVF013AKyuVMbMB7/0m4BCgt7iQ934eMC8pR6FQGGHYo6+9vV3xNljeYla8jaV4Gy/rMdeSoMq1hMIIymBm84H5g6/39vaWFsmsQqGA4m2svMWseBtL8TZeWjF3dHTUVK6WLr4uYHrR9DSgp1IZ7307cBCwoaYIREREyqilBbUEmOW9nwl0A+cC55eUuRO4ALgPeDvwUzPbqwUlIiJSq6otKDMbAC4C7gIeibNsqff+Ku/9OUmxrwKHeO+XA5cCn2pUwCIi0hpcCKk1dEJPT2lPYXblrX85b/FC/mJWvI2leBsv5XNQVUd660oSIiKSSUpQIiKSSUpQIiKSSameg0prxSIikrrsnoPy3v+eGGAuHopXMStexdtM8WYg5qrUxSciIpmkBCUiIpmUZoKaX71IpijexstbzIq3sRRv42U65jQHSYiIiFSkLj4REckkJSgREcmkWq5mXnfe+znA9UAbcLOZXZtCDNOBW4EjgN3AfDO73ns/BfhvYAbQCXgz25jcIfh64C3AduB9ZnZ/sqwLgMuTRX/WzL7WwLjbgN8B3WZ2dnKV+YXAFOB+4D1mttN7f0CyfScDfcA7zawzWcZlwIXALuCjZnZXA+OdDNwMnEj87dsHgMfIaB177/8e+GAS6x+B9wNHkpE69t7fApwNrDOzE5N5dfvMeu9PBhYAY4FFwMX7emeCCjF/AfgLYCfwBPB+M+tPXitbd5X2G5W+A/WMt+i1jwNfAA41s94s1HGleL33HyFe6HsA+L6ZfSKZn2r9Dseot6CSHewNwFnAbOA87/3s0Y6D+KZ9zMyOB04DPpzE8SngJ2Y2C/gJz1+Z/SxgVvKYB9wEz+0cPkO8y/CpwGe89wc3MO6LiVeVH/Q54Lok3o3EDx7J341mdixwXVKOZBvPBU4A5gA3Ju9Jo1wP/NDMXgS8NIk9k3XsvZ8KfBQ4JfmitxHrKkt1vCBZZrF61udNSdnB/ytdV71ivhs40cxeAjwOXJbEVrbuquw3Kr0/9Yx38KD2TcCqotlZqOO94vXevx6YC7zEzE4A/jmZn4X6rVkaXXynAsvNbEWShRcSK3JUmdmawSMdM9tC3HFOTWIZPDr/GvC25Plc4FYzC2b2G2Cy9/5I4M+Bu81sg5ltJH7x6vGl3ov3fhrwVmKLhOTo7Q3A7RXiHdyO24E3JuXnAgvN7BkzWwksJ74njYh3EvBa4u1YMLOdyVFyZuuY2KswNrnx5jhgDRmqYzP7BXvfDLQu9Zm8NsnM7kuO6G8tWlZdYzazHyW38gH4DfFGqIMxl6u7svuNKt+BusWbuA74BHteBSf1Oq4Q74eAa83smaTMuqJ4U63f4UgjQU0FVhdNdyXzUuO9nwGcBPwWONzM1kBMYsBhSbFKcY/m9vwr8QuyO5k+BOgv+qIXr/u5uJLXNyXlRzPeY4D1wH967//gvb/Zez+ejNaxmXUTjzRXERPTJuD3ZLuOoX71OTV5Xjq/0T4A/CB5PtyYh/oO1E1y77tuM3uw5KWs1vFxwGu897/13v/ce/+KEcY7KvVbSRoJqtwlLlJj5sZGAAAC/UlEQVQb6+69nwB8G7jEzDYPUbRS3KOyPd77wT7m39cQ01CvjWb9twMvB24ys5OAbQx9M8u06/hg4hHmTKADGE/s8qi07izU8VCGG9+ox+29/zSxu/22ZFbmYvbejwM+DVxR5uXMxZtoBw4mnr74B8CS1lBW4y0rjQTVBUwvmp4GpHLnQu/9fsTkdJuZfSeZ/VTSDCf5O9g0rhT3aG3Pq4BzvPedxOb3G4gtqslJd1Tpup+LK3n9IGI3wGjWfxfQZWa/TaZvJyasrNbxmcBKM1tvZs8C3wH+jGzXMdSvPrt4vquteH5DJIMIzgbeVTRIYLgx91L5/amXFxAPWh5Mvn/TgPu990eMIN7RquMu4DtJ1+NiYq9LYQTxjkb9VpRGgloCzPLez/Te7088YXfnaAeRHE18FXjEzP6l6KU7gQuS5xcA3yua/17vvfPenwZsSrpT7gLe7L0/ODkCf3Myr67M7DIzm2ZmM4h19lMzexdwD/D2CvEObsfbk/IhmX+u9/6AZHTOLGBxveNNYl4LrPbevzCZ9UbgYTJax8SuvdO89+OSz8dgvJmt4zJxjLg+k9e2eO9PS7b/vUXLqqtkxNgngXPMbHvJtpSru7L7jaS+K70/dWFmfzSzw8xsRvL96wJenny+s1rHdxAPYvHeHwfsT0w2mavfoYx6gkr6Mi8ivoGPxFm2dLTjILZI3gO8wXv/QPJ4C3At8Cbv/TLiiJ3BIfCLgBXEk4pfAf4XgJltAK4mvsFLgKuSeaPlk8Cl3vvlxP7irybzvwocksy/lKRrLalrI+54fwh82Mx2NTC+jwC3ee8fAl4GXENG6zhp6d1OHEr7R+L3Yz4ZqmPv/TeB+4AXeu+7vPcXUt/6/BBxEM5y4vDvwXND9Y75S8BE4O7ku/flJLaydVdlv1Hp/alnvJWkXscV4r0FOMZ7/ydij8sFSWsq9fodDl3qSEREMklXkhARkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUz6H3ak05Y7FQgqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.Series(sampling_table).plot(title='Skip-Gram Sampling Probabilities')\n",
    "plt.tight_layout();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Generate target-context word pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:00:33.846407Z",
     "start_time": "2018-12-10T06:58:10.973923Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17,324,656 pairs created\n"
     ]
    }
   ],
   "source": [
    "pairs, labels = skipgrams(sequence=data,\n",
    "                            vocabulary_size=vocab_size,\n",
    "                            window_size=WINDOW_SIZE,\n",
    "                            sampling_table=sampling_table,\n",
    "                            negative_samples=1.0,\n",
    "                            shuffle=True)\n",
    "\n",
    "print('{:,d} pairs created'.format(len(pairs)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:00:33.851520Z",
     "start_time": "2018-12-10T07:00:33.848200Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1636, 654], [12035, 9246], [217, 6031], [1522, 7], [3917, 1024]]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pairs[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:01:53.633534Z",
     "start_time": "2018-12-10T07:00:33.852713Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "target_word, context_word = np.array(pairs, dtype=np.int32).T\n",
    "labels = np.array(labels, dtype=np.int8)\n",
    "del pairs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:01:53.650930Z",
     "start_time": "2018-12-10T07:01:53.635164Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1636, 12035,   217,  1522,  3917], dtype=int32)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_word[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:01:53.661703Z",
     "start_time": "2018-12-10T07:01:53.652325Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>context</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1636</td>\n",
       "      <td>654</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12035</td>\n",
       "      <td>9246</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>217</td>\n",
       "      <td>6031</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1522</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3917</td>\n",
       "      <td>1024</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   target  context  label\n",
       "0    1636      654      0\n",
       "1   12035     9246      0\n",
       "2     217     6031      0\n",
       "3    1522        7      1\n",
       "4    3917     1024      1"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'target': target_word[:5], 'context': context_word[:5], 'label': labels[:5]})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:01:54.026465Z",
     "start_time": "2018-12-10T07:01:53.662752Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    8662328\n",
       "0    8662328\n",
       "dtype: int64"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(labels).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:05:58.136170Z",
     "start_time": "2018-12-10T07:05:58.102377Z"
    },
    "hide_input": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with pd.HDFStore(path / 'data.h5') as store:\n",
    "    store.put('id_to_token', pd.Series(id_to_token))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Define Keras Model Components"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Scalar Input Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:01.438878Z",
     "start_time": "2018-12-10T07:06:01.428723Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "input_target = Input((1,), name='target_input')\n",
    "input_context = Input((1,), name='context_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Shared Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:04.673449Z",
     "start_time": "2018-12-10T07:06:04.671203Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "embedding = Embedding(input_dim=vocab_size,\n",
    "                      output_dim=EMBEDDING_SIZE,\n",
    "                      input_length=1,\n",
    "                      name='embedding_layer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:05.431704Z",
     "start_time": "2018-12-10T07:06:05.387080Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "target = embedding(input_target)\n",
    "target = Reshape((EMBEDDING_SIZE, 1), name='target_embedding')(target)\n",
    "\n",
    "context = embedding(input_context)\n",
    "context = Reshape((EMBEDDING_SIZE, 1), name='context_embedding')(context)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Create Similarity Measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:06.339306Z",
     "start_time": "2018-12-10T07:06:06.314257Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "dot_product = Dot(axes=1)([target, context])\n",
    "dot_product = Reshape((1,), name='similarity')(dot_product)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Sigmoid Output Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:08.599771Z",
     "start_time": "2018-12-10T07:06:08.584301Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "output = Dense(units=1, activation='sigmoid', name='output')(dot_product)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Compile Training Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:09.195879Z",
     "start_time": "2018-12-10T07:06:09.117732Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "model = Model(inputs=[input_target, input_context], outputs=output)\n",
    "model.compile(loss='binary_crossentropy', optimizer='rmsprop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:06:11.692010Z",
     "start_time": "2018-12-10T07:06:11.689022Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "target_input (InputLayer)       (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "context_input (InputLayer)      (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_layer (Embedding)     (None, 1, 200)       3476200     target_input[0][0]               \n",
      "                                                                 context_input[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "target_embedding (Reshape)      (None, 200, 1)       0           embedding_layer[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "context_embedding (Reshape)     (None, 200, 1)       0           embedding_layer[1][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dot (Dot)                       (None, 1, 1)         0           target_embedding[0][0]           \n",
      "                                                                 context_embedding[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "similarity (Reshape)            (None, 1)            0           dot[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "output (Dense)                  (None, 1)            2           similarity[0][0]                 \n",
      "==================================================================================================\n",
      "Total params: 3,476,202\n",
      "Trainable params: 3,476,202\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Validation Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:01.364246Z",
     "start_time": "2018-12-10T07:13:01.330374Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "similarity = Dot(normalize=True, \n",
    "                 axes=1, \n",
    "                 name='cosine_similarity')([target, context])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:01.715914Z",
     "start_time": "2018-12-10T07:13:01.711419Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# create a secondary validation model to run our similarity checks during training\n",
    "validation_model = Model(inputs=[input_target, input_context], outputs=similarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:02.480815Z",
     "start_time": "2018-12-10T07:13:02.470139Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "target_input (InputLayer)       (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "context_input (InputLayer)      (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_layer (Embedding)     (None, 1, 200)       3476200     target_input[0][0]               \n",
      "                                                                 context_input[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "target_embedding (Reshape)      (None, 200, 1)       0           embedding_layer[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "context_embedding (Reshape)     (None, 200, 1)       0           embedding_layer[1][0]            \n",
      "__________________________________________________________________________________________________\n",
      "cosine_similarity (Dot)         (None, 1, 1)         0           target_embedding[0][0]           \n",
      "                                                                 context_embedding[0][0]          \n",
      "==================================================================================================\n",
      "Total params: 3,476,200\n",
      "Trainable params: 3,476,200\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "validation_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![Keras Graph](https://s3.amazonaws.com/applied-ai/images/keras_graph_tensorboard.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Create Keras Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "####  Nearest Neighors & Analogies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:07.729779Z",
     "start_time": "2018-12-10T07:13:07.726679Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_set = analogies_id.dropna().astype(int)\n",
    "a, b, c, actual = test_set.values.T\n",
    "actual = actual.reshape(-1, 1)\n",
    "n_analogies = len(actual)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:10.070985Z",
     "start_time": "2018-12-10T07:13:10.064607Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class EvalCallback(Callback):\n",
    "        \n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.eval_nn()\n",
    "        self.test_analogies()\n",
    "\n",
    "    def on_train_end(self, logs={}):\n",
    "        self.eval_nn()\n",
    "\n",
    "    def on_epoch_end(self, batch, logs={}):\n",
    "        self.test_analogies()\n",
    "\n",
    "    @staticmethod\n",
    "    def test_analogies():\n",
    "        print('\\nAnalogy Accuracy:\\n\\t', end='')\n",
    "        embeddings = embedding.get_weights()[0]\n",
    "        target = embeddings[c] + embeddings[b] - embeddings[a]\n",
    "        neighbors = np.argsort(cdist(target, embeddings, metric='cosine'))\n",
    "        match_id = np.argwhere(neighbors == actual)[:, 1]\n",
    "        print('\\n\\t'.join(['Top {}: {:.2%}'.format(i, (match_id < i).sum() / n_analogies) for i in [1, 5, 10]]))\n",
    "\n",
    "    def eval_nn(self):\n",
    "        print('\\n{} Nearest Neighbors:'.format(NN))\n",
    "        for i in range(VALID_SET):\n",
    "            valid_id = valid_examples[i]\n",
    "            valid_word = id_to_token[valid_id]\n",
    "            similarity = self._get_similiarity(valid_id).reshape(-1)\n",
    "            nearest = (-similarity).argsort()[1:NN + 1]\n",
    "            neighbors = [id_to_token[nearest[n]] for n in range(NN)]\n",
    "            print('{}:\\t{}'.format(valid_word, ', '.join(neighbors)))            \n",
    "        \n",
    "    @staticmethod\n",
    "    def _get_similiarity(valid_word_idx):\n",
    "        target = np.full(shape=vocab_size, fill_value=valid_word_idx)\n",
    "        context = np.arange(vocab_size)\n",
    "        return validation_model.predict([target, context])\n",
    "\n",
    "\n",
    "evaluation = EvalCallback()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Tensorboard Callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:13:13.272624Z",
     "start_time": "2018-12-10T07:13:13.270670Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "tensorboard = TensorBoard(log_dir=str(tb_path),\n",
    "                          write_graph=True,\n",
    "                          embeddings_freq=1,\n",
    "                          embeddings_metadata=str(tb_path / 'meta.tsv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:29:39.905357Z",
     "start_time": "2018-12-10T07:13:13.616616Z"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "loss = model.fit(x=[target_word, context_word],\n",
    "                 y=labels,\n",
    "                 shuffle=True,\n",
    "                 batch_size=BATCH_SIZE,\n",
    "                 epochs=EPOCHS,\n",
    "                 callbacks=[evaluation, tensorboard])\n",
    "\n",
    "model.save(str(path / 'skipgram_model.h5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Optimized TensorFlow Model\n",
    "\n",
    "Compile custom ops using `compile-ops.sh`.\n",
    "\n",
    "Run from command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mcompile-ops.sh\u001b[m\u001b[m      \u001b[31mword2vec.py\u001b[m\u001b[m         word2vec_ops.so.zip\r\n",
      "\u001b[31mrun_tf.sh\u001b[m\u001b[m           \u001b[31mword2vec_ops.so\u001b[m\u001b[m\r\n"
     ]
    }
   ],
   "source": [
    "!ls tensorflow/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# %%bash\n",
    "# python tensorflow/word2vec.py --language=en --source=Ted --file=ngrams_1 --embedding_size=300 --num_neg_samples=20 --starter_lr=.1 --target_lr=.05 --batch_size=10 --min_count=10 --window_size=10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## word2vec using Gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-07T00:33:47.261154Z",
     "start_time": "2018-12-07T00:33:47.256178Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def accuracy_by_category(acc, detail=True):\n",
    "    results = [[c['section'], len(c['correct']), len(c['incorrect'])] for c in acc]\n",
    "    results = pd.DataFrame(results, columns=['category', 'correct', 'incorrect'])\n",
    "    results['average'] = results.correct.div(results[['correct', 'incorrect']].sum(1))\n",
    "    if detail:\n",
    "        print(results.sort_values('average', ascending=False))\n",
    "    return results.loc[results.category=='total', ['correct', 'incorrect', 'average']].squeeze().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-07T00:33:48.336482Z",
     "start_time": "2018-12-07T00:33:48.331924Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "ANALOGIES_PATH = PROJECT_DIR / 'data' / 'analogies' / 'analogies-{}.txt'.format(LANGUAGE)\n",
    "gensim_path = PROJECT_DIR / 'gensim' / SOURCE / LANGUAGE / FILE_NAME\n",
    "if not gensim_path.exists():\n",
    "    gensim_path.mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Sentence Generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:30:26.875274Z",
     "start_time": "2018-12-10T07:30:26.871082Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sentence_path = PROJECT_DIR / 'vocab' / SOURCE / LANGUAGE / '{}.txt'.format(FILE_NAME)\n",
    "sentences = LineSentence(str(sentence_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-10T07:36:42.963953Z",
     "start_time": "2018-12-10T07:30:46.174109Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Duration: 367.1s\n",
      "                       category  correct  incorrect  average\n",
      "0      capital-common-countries        0        132     0.00\n",
      "1                 capital-world        0         12     0.00\n",
      "2                 city-in-state        0        178     0.00\n",
      "3                      currency        0         10     0.00\n",
      "4                        family        0        210     0.00\n",
      "5     gram1-adjective-to-adverb        0        600     0.00\n",
      "6                gram2-opposite        0        182     0.00\n",
      "7             gram3-comparative        0        930     0.00\n",
      "8             gram4-superlative        0        420     0.00\n",
      "9      gram5-present-participle        0        702     0.00\n",
      "10  gram6-nationality-adjective        0        552     0.00\n",
      "11             gram7-past-tense        0       1122     0.00\n",
      "12                 gram8-plural        0        870     0.00\n",
      "13           gram9-plural-verbs        0        506     0.00\n",
      "14                        total        0       6426     0.00\n",
      "Base Accuracy: Correct 0 | Wrong 6,426 | Avg 0.00%\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/ipykernel_launcher.py:17: DeprecationWarning: Call to deprecated `accuracy` (Method will be removed in 4.0.0, use self.evaluate_word_analogies() instead).\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "\n",
    "model = Word2Vec(sentences,\n",
    "                 sg=1,\n",
    "                 size=300,\n",
    "                 window=5,\n",
    "                 min_count=10,\n",
    "                 negative=10,\n",
    "                 workers=8,\n",
    "                 iter=20,\n",
    "                 alpha=0.05)\n",
    "\n",
    "model.wv.save(str(gensim_path / 'word_vectors.bin'))\n",
    "print('Duration: {:,.1f}s'.format(time() - start))\n",
    "\n",
    "# gensim computes accuracy based on source text files\n",
    "detailed_accuracy = model.wv.accuracy(str(ANALOGIES_PATH), case_insensitive=True)\n",
    "\n",
    "# get accuracy per category\n",
    "summary = accuracy_by_category(detailed_accuracy)\n",
    "print('Base Accuracy: Correct {:,.0f} | Wrong {:,.0f} | Avg {:,.2%}\\n'.format(*summary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-07T00:39:36.863411Z",
     "start_time": "2018-12-07T00:39:36.825185Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>token</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>managing</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dancer</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>she</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>republic</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>graduation</td>\n",
       "      <td>0.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>england</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>kentucky</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>junior</td>\n",
       "      <td>0.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>jordan</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>zoe</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        token  similarity\n",
       "0    managing        0.25\n",
       "1      dancer        0.25\n",
       "2         she        0.25\n",
       "3    republic        0.25\n",
       "4  graduation        0.25\n",
       "5     england        0.24\n",
       "6    kentucky        0.24\n",
       "7      junior        0.23\n",
       "8      jordan        0.22\n",
       "9         zoe        0.22"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_sim = model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)\n",
    "pd.DataFrame(most_sim, columns=['token', 'similarity'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-07T00:39:36.909259Z",
     "start_time": "2018-12-07T00:39:36.867047Z"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>'ve</th>\n",
       "      <td>have</td>\n",
       "      <td>’ve</td>\n",
       "      <td>had</td>\n",
       "      <td>been</td>\n",
       "      <td>'d</td>\n",
       "      <td>got</td>\n",
       "      <td>has</td>\n",
       "      <td>seen</td>\n",
       "      <td>'re</td>\n",
       "      <td>anthropology</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>for</th>\n",
       "      <td>to</td>\n",
       "      <td>in</td>\n",
       "      <td>and</td>\n",
       "      <td>of</td>\n",
       "      <td>at</td>\n",
       "      <td>is</td>\n",
       "      <td>the</td>\n",
       "      <td>with</td>\n",
       "      <td>so</td>\n",
       "      <td>that</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>work</th>\n",
       "      <td>working</td>\n",
       "      <td>works</td>\n",
       "      <td>worked</td>\n",
       "      <td>collaborate</td>\n",
       "      <td>interfere</td>\n",
       "      <td>partnerships</td>\n",
       "      <td>do</td>\n",
       "      <td>research</td>\n",
       "      <td>happen</td>\n",
       "      <td>resonate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>time</th>\n",
       "      <td>moment</td>\n",
       "      <td>night</td>\n",
       "      <td>periods</td>\n",
       "      <td>year</td>\n",
       "      <td>lapse</td>\n",
       "      <td>seconds</td>\n",
       "      <td>slept</td>\n",
       "      <td>tense</td>\n",
       "      <td>sabbatical</td>\n",
       "      <td>lifetimes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>'m</th>\n",
       "      <td>am</td>\n",
       "      <td>'re</td>\n",
       "      <td>’m</td>\n",
       "      <td>i</td>\n",
       "      <td>optimist</td>\n",
       "      <td>woo</td>\n",
       "      <td>want</td>\n",
       "      <td>was</td>\n",
       "      <td>technologist</td>\n",
       "      <td>bruno_giussani</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>at</th>\n",
       "      <td>the</td>\n",
       "      <td>in</td>\n",
       "      <td>and</td>\n",
       "      <td>archives</td>\n",
       "      <td>for</td>\n",
       "      <td>on</td>\n",
       "      <td>from</td>\n",
       "      <td>della</td>\n",
       "      <td>fireman</td>\n",
       "      <td>sachs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>had</th>\n",
       "      <td>have</td>\n",
       "      <td>was</td>\n",
       "      <td>got</td>\n",
       "      <td>has</td>\n",
       "      <td>could</td>\n",
       "      <td>'d</td>\n",
       "      <td>'ve</td>\n",
       "      <td>were</td>\n",
       "      <td>would</td>\n",
       "      <td>wanted</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>put</th>\n",
       "      <td>putting</td>\n",
       "      <td>squeeze</td>\n",
       "      <td>bring</td>\n",
       "      <td>puts</td>\n",
       "      <td>take</td>\n",
       "      <td>strap</td>\n",
       "      <td>sew</td>\n",
       "      <td>taped</td>\n",
       "      <td>hold</td>\n",
       "      <td>throw</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>your</th>\n",
       "      <td>my</td>\n",
       "      <td>you</td>\n",
       "      <td>their</td>\n",
       "      <td>our</td>\n",
       "      <td>her</td>\n",
       "      <td>his</td>\n",
       "      <td>purse</td>\n",
       "      <td>yourself</td>\n",
       "      <td>wallet</td>\n",
       "      <td>the</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>things</th>\n",
       "      <td>ways</td>\n",
       "      <td>thing</td>\n",
       "      <td>modes</td>\n",
       "      <td>distractions</td>\n",
       "      <td>attributes</td>\n",
       "      <td>something</td>\n",
       "      <td>technologies</td>\n",
       "      <td>projects</td>\n",
       "      <td>places</td>\n",
       "      <td>sorts</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0        1        2             3           4             5  \\\n",
       "'ve        have      ’ve      had          been          'd           got   \n",
       "for          to       in      and            of          at            is   \n",
       "work    working    works   worked   collaborate   interfere  partnerships   \n",
       "time     moment    night  periods          year       lapse       seconds   \n",
       "'m           am      're       ’m             i    optimist           woo   \n",
       "at          the       in      and      archives         for            on   \n",
       "had        have      was      got           has       could            'd   \n",
       "put     putting  squeeze    bring          puts        take         strap   \n",
       "your         my      you    their           our         her           his   \n",
       "things     ways    thing    modes  distractions  attributes     something   \n",
       "\n",
       "                   6         7             8               9  \n",
       "'ve              has      seen           're    anthropology  \n",
       "for              the      with            so            that  \n",
       "work              do  research        happen        resonate  \n",
       "time           slept     tense    sabbatical       lifetimes  \n",
       "'m              want       was  technologist  bruno_giussani  \n",
       "at              from     della       fireman           sachs  \n",
       "had              've      were         would          wanted  \n",
       "put              sew     taped          hold           throw  \n",
       "your           purse  yourself        wallet             the  \n",
       "things  technologies  projects        places           sorts  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "similars = pd.DataFrame()\n",
    "for id in valid_examples:\n",
    "    word = id_to_token[id]\n",
    "    similars[word] = [s[0] for s in model.wv.most_similar(id_to_token[id])]\n",
    "    \n",
    "similars.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Continue Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-12-07T00:45:39.510380Z",
     "start_time": "2018-12-07T00:39:36.914176Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/stefan/.pyenv/versions/at-3.6/lib/python3.6/site-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `accuracy` (Method will be removed in 4.0.0, use self.evaluate_word_analogies() instead).\n",
      "  \"\"\"\n",
      "/home/stefan/.pyenv/versions/at-3.6/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 | Duration: 36.0 | Accuracy: 8.03% \n",
      "2 | Duration: 37.1 | Accuracy: 8.69% \n",
      "3 | Duration: 36.1 | Accuracy: 8.91% \n",
      "4 | Duration: 36.1 | Accuracy: 8.43% \n",
      "5 | Duration: 36.5 | Accuracy: 8.52% \n",
      "6 | Duration: 36.3 | Accuracy: 8.77% \n",
      "7 | Duration: 36.0 | Accuracy: 8.96% \n",
      "8 | Duration: 36.1 | Accuracy: 8.64% \n",
      "9 | Duration: 36.4 | Accuracy: 8.88% \n",
      "10 | Duration: 35.9 | Accuracy: 8.57% \n"
     ]
    }
   ],
   "source": [
    "accuracies = [summary]\n",
    "for i in range(1, 11):\n",
    "    start = time()\n",
    "    model.train(sentences, epochs=1, total_examples=model.corpus_count)\n",
    "    detailed_accuracy = model.wv.accuracy(str(ANALOGIES_PATH))\n",
    "    accuracies.append(accuracy_by_category(detailed_accuracy, detail=False))\n",
    "    print('{} | Duration: {:,.1f} | Accuracy: {:.2%} '.format(i, time() - start, accuracies[-1][-1]))\n",
    "\n",
    "pd.DataFrame(accuracies, columns=['correct', 'wrong', 'average']).to_csv(gensim_path / 'accuracies.csv', index=False)\n",
    "model.wv.save(str(gensim_path / 'word_vectors_final.bin'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The `google` command-line Tool"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Run from Command Line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "file_name=../data/wiki/en/wiki.txt\n",
    "time ./word2vec -train \"$file_name\" -output vectors_en.bin - cbow 1 -size 300  -min-count 10  -window 10 -negative 10 -hs 0 -sample 1e-4 -threads 8 -binary 1 -iter 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Load Trained Model & Word Vectors via `gensim`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "file_name = 'word2vec/word_vectors/vectors_en.bin'\n",
    "model = KeyedVectors.load_word2vec_format(file_name, binary=True, unicode_errors='ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100000, 300)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectors = model.vectors[:100000]\n",
    "vectors /= norm(vectors, axis=1).reshape(-1, 1)\n",
    "vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "words = model.index2word[:100000]\n",
    "word2id = {w:i for i, w in enumerate(words)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Compute Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "analogy_path = PROJECT_DIR / 'data/analogies/analogies-en.txt'\n",
    "accuracy = model.accuracy(questions=str(analogy_path), restrict_vocab=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                       category  correct  incorrect  average\n",
      "0      capital-common-countries      459         47     0.91\n",
      "7             gram3-comparative     1207        125     0.91\n",
      "10  gram6-nationality-adjective     1472        168     0.90\n",
      "4                        family      438         68     0.87\n",
      "1                 capital-world     7058       1132     0.86\n",
      "12                 gram8-plural     1082        250     0.81\n",
      "14                        total    17974       6458     0.74\n",
      "11             gram7-past-tense     1118        442     0.72\n",
      "9      gram5-present-participle      726        330     0.69\n",
      "13           gram9-plural-verbs      580        290     0.67\n",
      "2                 city-in-state     2601       1641     0.61\n",
      "8             gram4-superlative      605        387     0.61\n",
      "6                gram2-opposite      254        502     0.34\n",
      "5     gram1-adjective-to-adverb      312        680     0.31\n",
      "3                      currency       62        396     0.14\n",
      "\n",
      "Overall Accuracy: Correct 17,974 | Wrong 6,458 | Avg 73.57%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "summary = accuracy_by_category(accuracy, detail=True)\n",
    "print('\\nOverall Accuracy: Correct {:,.0f} | Wrong {:,.0f} | Avg {:,.2%}\\n'.format(*summary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Project Data using `tensorboard` Projector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "PROJECTION_LIMIT = 10000\n",
    "proj_path = Path('word2vec', 'projector')\n",
    "pd.Series(words).iloc[:PROJECTION_LIMIT].to_csv(proj_path / 'meta_data.tsv', index=False, header=None, sep='\\t')\n",
    "pd.DataFrame(vectors).iloc[:PROJECTION_LIMIT].to_csv(proj_path / 'embeddings.tsv', index=False, header=None, sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Project Analogies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Incremental PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   3.69\n",
       "1   3.19\n",
       "dtype: float64"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca = IncrementalPCA(n_components=2)\n",
    "\n",
    "vectors2D = pca.fit_transform(vectors)\n",
    "pd.Series(pca.explained_variance_ratio_).mul(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Group Analogies by Category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "results = pd.DataFrame()\n",
    "correct = incorrect = 0\n",
    "for section in accuracy:\n",
    "    correct += len(section['correct'])\n",
    "    incorrect += len(section['incorrect'])\n",
    "    df = pd.DataFrame(section['correct']).apply(lambda x: x.str.lower()).assign(section=section['section'])\n",
    "    results = pd.concat([results, df])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Identify Analogy most similar in 2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def find_most_similar_analogy(v):\n",
    "    \"\"\"Find analogy that most similar in 2D\"\"\"\n",
    "    v1 = vectors2D[v[1]] - vectors2D[v[0]]\n",
    "    v2 = vectors2D[v[3]] - vectors2D[v[2]]\n",
    "    idx, most_similar = None, np.inf\n",
    "    \n",
    "    for i in range(len(v1)):\n",
    "        similarity = cosine(v1[i], v2[i])\n",
    "        if similarity < most_similar:\n",
    "            idx = i\n",
    "            most_similar = similarity\n",
    "    return idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def get_plot_lims(coordinates):\n",
    "    xlim, ylim = coordinates.agg(['min', 'max']).T.values\n",
    "    xrange, yrange = (xlim[1] - xlim[0]) * .1, (ylim[1] - ylim[0]) * .1\n",
    "    xlim[0], xlim[1] = xlim[0] - xrange, xlim[1] + xrange\n",
    "    ylim[0], ylim[1] = ylim[0] - yrange, ylim[1] + yrange\n",
    "    return xlim, ylim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJYAAAKACAYAAADOyPLPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XecHXX1//HXIfQOShOBgCK9G6QTqoB08FBECCKKFAFZEZASFBB1KV8LEooElXYApUhHCR1CF34UkSoI0jtIyfn98Zk1N+tms0l2d+7ufT8fj31k79y5c8/c7M7OnDmf87HMREREREREREREZHJNU3cAIiIiIiIiIiIyMCmxJCIiIiIiIiIiU0SJJRERERERERERmSJKLImIiIiIiIiIyBRRYklERERERERERKaIEksiIiIiIiIiIjJFlFjqATNbzswuM7M3zew1M7vKzFbs5fcYbWZXd1o23WRuY6SZjZnEOn2+L82umT6DnvwfV/+vj/ZHPCJTazAcL83s/8zs3k7LZjOzj8xsq07LjzazmJz37uL90syGd/P8CDN7emreQ6TVTOpY1BvHkYbXDa1+j1ebininNbODzewRM3vPzP5pZr82s7mmNkYrpp3S2ESkbzTTNYnI1FJiaRLMbCXgNuB5YAlgceBB4AYz+3xvvU9mjsjMTRre93rgh721/Wqb/bIvzayZPgMzOxy4dlLrZebIzFyyH0ISmSqD6Hh5I7CCmc3esGxtYFpg407rrg7c0ovvLSJTqSfHov4475pMPwH2BfYGPgVsTon7WjMbAmBmawMfTsG2Hwd26aU4RaQXNNM1iUhvUGJp0k4BHgX2zswXM/PVzDwYuJe+PQHpiztLde1LM2mmz2BawPr5PUX60mA5Xt5E+d1cs2HZesCrNCSWzGwa4EsosSTSbKbkWFR3Rc+uwPGZeUNmvp+ZDwBbAB8AS1XrDJnCbde9byLyv5rpmkRkqimx1A0z+xywGnBGZmanp4+kZJkxs8+Y2blm9qKZvW9mj5nZbg3b6SiRXtrM7jCzD8xsrJmt0LDO6OprSTNLYF3gqOp1G1brbGFmt1flkm+Y2XVm9oXe3Jdq3T3N7HEz+4+Z/T8z26bTvjxnZqua2f1m9o6ZnW1m05nZblXp9utmNrLT+2e1b7eY2btm9mczm8PMNq3e620z+01jqbaZzWlmv61KQ98xs0vMbKGG50ea2flmtouZPWNmb5nZ6WY2Yz98BmlmQye2zMrQlTvMbOOqrP1dM/tjR0m7mb0IHAWsW73umGr5aDNrszK85i0z29A6DdmpPuvjzOz56nO70cxWaXh+mJndVH1mr1Sf0bxdfSYivWUwHS8z8xXgYWCdhsXrAT8FPt/wu78MJQH1QPWe05jZYWb2bBX3XVYqDDriHm5mD5nZGmb2dzP7/UQ+y6+a2T+qz+cSSvWCiPTAZByLuj2OmNmjZrZXp21H4/GqmxiGV8e4lc3sHitD2/5qZot087IhwKpm9t8bTpn5n8xcOzMfMrPzgRuq7aeZ3VJ9P5uZnWrl3OwDM3vKzA6untuk2rdFgLOq132+em7+6lj8ppm9VH0/36T2TUSmXk+OU53P/6vXdb4mGGNmO1THgPfN7PPVOr8ys72q64BvVuuuWl0zvG9mT5rZMWY2fcO2njazbczszOoa4ikz27HT+69mZndW2/i7me1SLT+1OkY1rru3mZ019Z+WDBRKLHVvuerf/9f5icy8NTNPrx5eDIwDVgbmAA4ARpnZop1edhKwD/BZykXLn61TEiQzH81MowzFODozLTOvN7OlgQuBXwALAIsB/wBG9ea+mNl3gOOrOOcBTgAusAn7BswCHAxsCQwDNgB+T7m7vwKwA3CYmW3U6a1+AnwLWLL6DH4HjKAMMRlevW73Ko5pgKuABYGVgC9QPuMrrCoJr6wKbAOsUe3jl4CD+uEz6ImFge9TPqehwGzAz6r3mx84Grix+j8+vOF1WwOLAktm5vVdbPc0yme2ITAvcAZwvZktUH02fwauqZ5bAZgBOL+L7Yj0psF0vKTa5tpQktzAssCpwAvAl6t11gDuyMxPqsc/Ab4JfBWYH7gEuKbTxeQslOPL7pTj3wTMbDngXODnwHyU3/fvT0bcIq2up8eijmVdHkeAC4D/XlRZGRq7IfDHHsYxM+U49k3Kceh5yt/riRkF7AY8ZGY/NbPNzWzWhjh3pCS4qWJcq3rqNOBzlET4bMBOwA/MbN3MvLrat2eA3avX/aM6lv4VeINyfrUE8G/KcVaV1CJ9b7KOU5OwH+V3eaHM/Ee1bGVgK+BLmXmGmS0FXA/8gXJ9sAHluu0nnbZ1PHA75Zh1IHB2xzmMmc1Dub64tnp+F+BEM1ufcrzcwsxmadjW1yjnM9IilFjqXscf9Jcmsd6ewHcy81+Z+WFmXkU5UKzSab0TM/Oe6m74fsCclAuQnngWGJaZ52Xme5n5GqWEclgPX9/TfTkc+FlmXpuZb2Xmb4EAjmhYZ07gyMx8NjMfqZ7fANg3M1/LzGuB+ykJoUanZ+bDmflP4LeUEu99qvLPeygHvI7XbEBJGo3IzGcy81+UC7HFgG0btvlpysnS85n5DHAmJbnU159BTywA7JmZj2fmy8Cvuomt0ULAHtU+T6C6+N4N2CszH6nK5X8P3AV8m3JSOS9wX/Vz8jywM+WPi0hfGkzHSygXmcOqC7B1gHsz823gOsYPh/tvfyUzmw3YHzg0M+/MzDcy81hKSfvBDdsdChxbnTh+wv/6HnB9Zo6qjj9XApNzginS6np6LJqU84G1zewz1eNtgb9Ux4GemA04KjPvy8w3gXa6Pwc4HNgLeJ1yHLgceLGqKuhuCNyRwE6Z+WRmfpSZdwBj6P54tyOlEnKfzPx3Zr5OSWAvQ6ncEpG+1VvHKYBpMvOQ6nypw/LA1zLzierxD4BrMvP0zHw7M5+iDLfbt7FqCbg9M8+o1rmEcmOv4/xsb+BFyjXgG5k5lpKYWp5yzvQm5dqu43plUUoCW1qEEkvde636d1LDEF6nZGyfqMqQk5IpnqXTeo91fFOdmPwdWLongWTmO8BCVoZzvGxmHwN/6+I9JmaS+2JluNRnKBdCje6iVL40apyl7C3g75nZ2FDyPWCmSbzmhU4HwcbXrAS8mJkvdDxZnZg93imWRzLzrYbHbzL+YN1Zb38Gk/JSZj7dw9gajc3Mjyby3BcpQ28eqkras/p52wj4Qma+QUlgXWZlGM5oyh2DcZMZu8jkGkzHSyh9lqanXAiux/iTo2uB9asLvTUY319pKUp1YE+OHd31ZFoOuKfTsie6WlFEutTTY1G3qhtnD1GqqWHK7r7f2fB9t+cAWYyqKpHmBr5CSWT/kHLjaGJeBn5oZejee9UxdVu6P94No9yEGtdwHvER5RysR0OGRWSq9MpxqnJrF8seqW6qdRgGbN/p2uFWynnO0Ib1Go9ZMOFxa3ng7szxQ/cy86TMPDkzx1EqxXeqntoZuHAiN9BkkFJiqXv3Akm5gzMBK+PvD7PSE+h6yh/irwPzVmXHdzDpxszT0MMLfjNbF7iUUp74JUqJ9eTMFDbJfQHeb4hrglWAzkNQOsfdVSKk8/5Pzmve7yKOrmJ5v4t1JvZz3aufQSddTf87ObE1ereb5z6gfI5DqpL2xq+dATJzP8rF6RmUZN1I4P6qokKkrwym4yWZ+SIlubUOJbF0Q/XUdZQhfJtRTsbuqJZPzvGzu9/xaSeyDRHpmZ78re+pC4AdzWwByg2vKycnkMzsfB7Q5e+ylf5sn2143duZeWVmbkOpBNikq9dVLqLcXNoXWLA6pp4/sfeqfMD4ofidv07rwa6JyNSZ0uNUV9cbXZ1TdF72AeOH+nb++nvDet1duxjdH1cuADap2gdoGFwLUmKpG9WFxTXAXl2MOT+EMmRpWcoFy96ZeVtmvlX9Qv3PgYLxs3pQjUFdDHhkIm/fOcO7DXB/Zh5ZlTt/CKzVxeumeF+qqoCn+N8hbMOAu3v6Xr3kAWC+hhL0jj4ni09pLL34Gbxe/Ttnw/M9/r9oMCVZ/AcpB/UJKiDMbDszW8nM5jOzzaphcqMyc29K7J+fwhhFemQwHS8b3EjpebYEVcPfzHyJcnw6snqP96p1/0FJ5E7t8fNhYMVOyya3WlKkZfXwWNSVrv4mn0/5HT4MuDQz/9NrgU5oQ+BRM5u/i+fepFR5Q6cYq+PnBsAPMvP6zHzdzKajtBJo1Hnf/gYsU63bsS0zs30mEoOI9KIeHqdeZ8JrDZjyc/m/0en8pOrNutckhto2eohSYd64jcPM7HvVw9spfSh/DEyfmZ2rn2SQU2Jp0g6kXJSfYmbzVF/tlHLA44DnKNnd7c1sFjNbEjiHcjDo/It6rJUZu+YGfkk5UbhwIu/7JLCGmc1a9fj4O7C4lY7+s5jZ1pSGkEzGAWFS+wJlrGybmW1kZaaREcD2wLE9fI9ekZk3US7kzjSzhau7hWdQPpeeNs7sylR/BtWQvAeAg8xsdjNbg9L3aHI9CSxlZguZWedhg12qhtadDZxhZquY2cxmtitwFiXhNCtwkZntb2Zzm9nMlF4K/6FcsIr0pcF0vISSWFqZMjy18S7etZRhqf8d0lY9f3IV96pWZr08hJIUOmEy3vNEYGMz27U6/uxESW6JSM/15G99Z52PI2Tmk5TE8D704d33LL0pLweuM7P1q+PWp8xsH0pPt19Wqz4NpJUZZ6cH3qY03d6qOh9ZhHKO8AkTHlOfpMxCO2O1b+dRhuKcVZ1jzUU59rQxPoklIn1rUsepm4BlzWyr6phwOGUI65T4CeXc4vtWZt1ejHL+tf5kDFf7DaXNwJHVOc7alN5sd0EZzkvpSbsP5RgjLUaJpUnIzEcpMwMNpfxhfoRSNbNOZv6z6hG0I2UM/quUkuRTgPsovXoaHVU99wLlDv3m3dz9Oo6SJHiF0sH/NMosan+mnETsxvgZhTq/zxTtS7XO6ZSZy35b7c9+wJaZeXNP3qOXbU8p5XyY0p/pI2DDTr2cJksvfgZfr173ImX2pm9NQTgXUvq2PEyZsaWn9qZc0F5J6a3wTWCzzLy3atL3ZUrFxtOUn7Vtq+cn5z1EJttgOl5Wbqz+vaHT8murfzv3SjqKkvjueN/NgPUy8/GevmFm3l3F+yNKU88dKE03RaSHevK3vgudjyMdzqf8re98HOhtu1AmIDmJcvx4mNJnae2OO/9V7IdQLgjfpFQ3bktJgP+bck5xHeX8oPFYdwilAe+blMkDPqScK8xCmTzhiWr9dRqqMEWkD/XgnOk+ysy5vwb+xfiZqqfkvf4ObApsRzlW3ERJmn99MrbxAuWYtE21jTMoEwA0Xh91DMPVMLgWZA39t6SPmNlQyvCqRTs1cxYRkQY6XopIM6kqD+fLzAPrjkVEpJmZ2WrAbzKzc1sAaQHT1h2AiIiIiEgzqXq7fYbSFHvjmsMREWlaVeuNGSj9lUbVHI7UREPhREREREQmNAYYCxyfmepRKCIyccdQ+mg+D5xecyxSEw2FExERERERERGRKaKKJRERERERERERmSJKLImIiIiIiIiIyBRRYklERERERERERKaIEksiIiIiIiIiIjJFlFgSEREREREREZEposSSiIiIiIiIiIhMESWWRERERERERERkiiixJCIiIiIiIiIiU0SJJRERERERaTlmlmZ2eadl05rZS2Y2ZjK3NcLMRlbfn2xmq/RepCIizW3augMQERERERGpwb+AL5rZfJn572rZpsALU7PRzDxgqiMTERlAVLEkIiIiIiKt6CPgQuBrDcu+Dpzd8cDMvmBm15vZDWZ2v5kd1PDcd8zsXjO7DFi5YfloMxtuZnOZ2RVmNtbM7jOzb1XPT2NmPzWzm6vndquWDzWzP5vZqWZ2i5ldbWaz9/FnICIy1QZVYqndbGS72TF1xyEiA1u7WbabTdvweHj7ZJbET2L7T/fWtkRERGSqnA10JHbmBBYF7m14fnngkMxcDxgGfNPMZjCzpYEDgXWArYAFutj2bsCLmbkqsDawsJlNA+wBfDYz1wY2AA4zs2Wq16wP/Coz1wIeAHbq1b2VHqnOBW9sN7uh3ezOdrNv1x2TSDMbVIklERERERGRnsrMewAzs5UABy7otMpfgd3N7FbgOuAzwLyUhNBVmflOZiZwdRebvw34ipmdSamEGpWZ44BNgIuq938buBLYuHrNQ5n5UPX9I9V7ST02aCsJxQ2Ag9tLMlFEujAoeyy1m+0JjAA+AWYA9mzL/Ft7aaj3EeXOwnzA79syT2g3mwUYBSwEzAg8Bezcljmu3ex+4LfANsBswO5tmQ/28y6JSJNoL3cafwKsQTm+/Lot8+x2s6HAr4DngGWBdwBvy3yr3WwJynHkXeD6hm2NAIa2ZY6sHo8GRrdljqmqmn4DbEk5Zp0ErAQMAd4Ctq+2dwLl7uYnwB3Ad9syP+mr/RcRERmEzqZcOywP7Aws0fDcKEqCZ3hmfmRm91D+Fne+QW+dN5qZY81sKWAt4IvA3Wa2zkTW7/jb/V6n5UMmb1ekt7VlvtNe/t+Xbzf7B/BzYDnKteEZbZmj2s1WBk4BkvKzcUBb5u3tZp+mnM/NS7n2PqAt865adkSkDw3GiiUDFgTWa8tcBzgUaGt4flVgM8pF4Z7t5e7E54Er2jLXbcv8EvAhMLxafzbg/SpbfQRwWL/shYjU7S/tZmOqIXAnNyzfA/hsW0P5enun8vW2/y1fPxv4cVvmxpST056YF/ioLXNNyjHtsbbMtdsy1wDGUu6qrgBs2Za5YlvmKsDTlDupIiIi0nPnALsA72Vm58bdnwOur5JKawMdVStjgI3MbEYrw+e37rxRKze7v5yZl2fmUcD9wBeAa4Ftq3VmoVQrXdP7uyW9od1sAWA14C7gYOCZtsz1gTWBEVUl05GUm42rU84Vl69e/n/AdW2Z6wLfAc5vN5u+v/dBpK8NxoqlBG4ALmo3mxuYjlI50OGS6m7+e+1mV1IuBE8ENm03u6l6/VBKqSuUuwTnVN8/AswbEdNR/gDsAXzL3V/p210SkRps0Jb5MZQeS8DIavkmwB8A2jLfro4jGwN/Ah5qm7B8faF2s9mApdoyO0rkrwR+2YP3n4lS5URb5rPtZh+2m/2Vctyer4rhIiDbS9PQW4Dr2zL/OeW7LCIi0noy80Uzux34fRdPHwL8ysxeA+6hXGfMlZn3mdlpwK3Am5Sq4c4uBc4ws/0ooyLGAldVzy1rZrcB44CRmfmYlepnaR5/aTfL6vt92zKfaDfbEqDjX2AWSrLxeuDYdrMVKT8n51bPbwLsC1CNoHkLWAa4r5/2QaRfDLbEklGGugXwlbbMu9vNlmPCi7jOZafjgL2BdSl3/t9oNzuBhrLTtsz3ImKGeQ4++CtvXHDBCsDfKI39/g283Xe7IyJNanLK17srlR/X6fGMndZ9B6Dd7CuU49SmbZkvtpcT1E9Vw+yWoVRgrgSc1252XFvm2Yh0YuXkuOMGCsBrmbmtlaq8DShl/Xtk5n5WhmkOycwzawlWRKQfZObQhu83b/h+DNXohcy8lvHVJ51ffzITVjV3LB/R8HDLzs9X9uvidU8zftQEmTl6Iq+V/vHfm4wNhgDfacsc23nldrPLKedkw4GRDT2ZJnbeKDJoDLbE0jKUpM9bwL3tZgZ8q9M627ebnU2pZNoY2JVS+np7lVSaj1LK2rmP0nem++xnj55mlllmBz5VLZsGOCcinqA05/sb8Ky762AhMnh1lK//sX18+fq2E1u5LfPNdrMn2s3Wbcu8kTKEbVz19LPAFgDtZosB6wGndrGZzwH3VUmlWSn9H65pNxsGbNuWeSgwpt1sRiZy8itS2SA7nSRn5vDq2/upLnR0MSMiItKlqynXjmPbzaajVKXtA/wAOLEt87x2sz8BL1Cqma4FtgNOr24GzkjP2yKIDBgDPrFUXdj9CZgZeAw4hpL4uRt4ndLbZJ2GlzxFGcP8aeCstsx7281epdzp35BShXQhMFfj+7j7yWftv/8r+eGHo6r3glIdtQxl+tDvURrpvhIRrwKvAi9RSiHvAB5z97d6efdFpP+dCSzb3lC+3pb5WHv35eu7AaPazT6mlNC/Vi2/Edij2ta/gD9O5PVnA1u0lxlp3qAc8xakTIe8c7vZWOBjSlL9G1Oxb9KCzOzpzBxaDcEYnZnDq4qloVk1lhcREREAjgV+0V6GTo6jNO9+qt3sLOCs6lxvNuCw6ubigcCZ7Wa7UM7VvC3zo9qiF+kjVmbHbA3VrHBPt03FndiImINykbcB8GdK5cCngaUoDdyWBeahJLc6vqanJJo6vl6hJLhupVQ5PaMqJxER6UtdDIXbLzMfVGJJRERERKbGgK9Y6m/u/iawdUTsDbzh7gm8XH3d1LhuREwPLAasQplJ4LOURNPSlCqnAyhVTq9GxCuMr3K6D7idUuX0Zn/sl4iItIT/GQonIiIiIjI1WqpiqZlExKcpCabVKT1RPk1JOnX821WV09OUKqcHUJWTiIhMhqpiabrOiSVVLImIiIjI1FBiqck0VDmtRJlVoKPKqSPhNDdVlRMl2fQqpVqqo8rpUVU5iYhIZ0osiYiINI+I+CplZvL/NHx9AFzi7o/VGZvI5NJQuCbj7h8Cj1Zf5zU+V1U5LUWpclqO0stpnmrZNsAMwGsNw+peZcIqp6dV5SQiIiIiIlK7fwNfBeZtWPYqpY+vyICiiqVBICKmo1Q5rUipclqI8RVOHV/vMb7CqaOX0/2Mr3J6o/8jF2ldEbEl8Ii7P153LCIAZrYC8OPM3LLuWERERFpBRPyc0ne3o+DjQWAtzSYuA40SS4NcRHyK8VVOK/C/vZxmZMI+Tq8CzwC3URJPT7u7Gr2K9LKIuAwYCqzn7q/WHI60ODM7DRgGHJKZ19Qdj4iISCuIiCHAGGAtSgXTjJQCgIuBH7n7+/VFJ9JzSiy1qKrKaVHGVzktzIRVTnMD7/O/vZweoCSdHnX31/s/cpHBISLuAr5IqRpc390/qDkkEREREelnEbEQcDNwC3AKcBxlVvFngT8A7e7+UX0RikyaEkvyPyJibmBJSsJpecq438ZhdTMBrzHh0LpnKBfI9wFPqcpJZOIiYg5KqfNC1aIrgC3dfVx9UYmIiIhIHSJic2Csu79UPd4UOJJyLfY4MAoYpXNFaVZKLEmPRcS0lCqnlShD6xZh/LC6uavvP2DChNMrjO/l9Ii7v9b/kYs0l4j4MqUxY8d4+o+AM939O/VFJSIiIiLNIiIM2An4HqW1ySNAO3CBu+siXpqKEkvSKxqqnDp6Oc3DhEPrOqqcXmXiVU4q8ZSWEBG/AnYG3gXGAT8DLnL3f9camIiIiIg0lYiYBtgb2BP4HPA34Gh3V09EaRpKLEmfaqhyWoEytG4R/nfGug+YMOH0MuWAeZa7v1xD2CJ9KiJWAN4BdqecKKzi7k/VG5WIiIiINKuqR+7BwNco7RTuBg519ztqDUwEJZakRhExF6XKaTXK8Lp5GD+0bh5gd3e/qL4IRfpWRCwN3Akc4+4/rTseERGRZlFVaRwBrAdsqP6dIkVEzASMBLalXDfdARzk7g/XGZe0NiWWpOlUVU5DgWc0PE4Gs2rs/IPA0+6+ed3xSGuLiFmAxdz9wbpjEZHWFhGLA2dRKt7vAjZ39/fqjUqkuVSTwRwPbEZpO3ILcKC7P1NrYNKSlFgSEalRRFxOGS66nBoxSp0iYkvgVGBtd3+i7nhEpPVUVUpHAnsAnwC/pky1rr+PIhMREfMCJwAbVIv+Qqlgeqm+qKTVTFN3ACK9zcwuNbMxZvaimT1WfX/MRNYdamZj+jlEkUY3USr0lqk5DpHNgQWAP1ZDlUVE+k1EfAG4GTgI+Dslyf1zJZVEuufuL7n71ymTKN0ObAKMjYjfVFVNIn1OiSUZdDJzq8wcDlwN/Dwzh2fm4TWHJTIxAXxIacQoUqfFqn+XBy6NiOnrDEZEWkNETBMRPwL+CnwGOIrSU+mf9UYmMrC4+zPuvh2wLvAIsBNwd0T8tOrLJNJnlFiSlmBm05jZT83sZjMba2a7dbHOvGZ2q5l9zsweN7NPVcsXMrMLq++/V61zp5kd2t/7IYNPNQ7+aWCVmkORFlb1+1qwYdFalKSniEifiYglKVVKBwKPAWu5+4mqUhKZcu7+sLtvCmwKPEeZgfieiPhhNbOcSK9TYklaxR7AZzNzbcr448PM7L9Dj8xsFuACYP/MfAK4GPhq9fQuwLlmthGwFeUuwNrAZma2ST/ugwxeTwGLVI3rReqwOKVS4G3gA0p/hlurhJOISK+KiCERcQzlWPMZSl+lDd39+XojExk83P12d18P2B54AzgUuCsi9qn6mYn0Gv1ASavYBLgIIDPfBq4ENq6emxb4HTALcG+17FzGD036CnBFtY1LMvPjzPyw2t6m/RK9DHZXAQsDq9YdiLS0k4BvV99fpN4mItIXImIpSpXSAZThOmu5+0k63oj0DXe/BlgT+CbwEfAzSg+mnXUDSXqLEkvSSjofOD+p/l0N+D1lbP/XATLzb8AcZrY98FCVSOpuGyJT41LgTWDnugOR1uTuf3f3kcCFwLOMT7yLiPSKqkrpOOB6YH7gMGAjVSmJ9D13T3c/H/gS8H1gOuB04LaI0I1ymWpKLEmruBbYFv477G1j4Jrqudsy8xJK9v4gM5uhWn4ucBol6dSxjS3NbFozm7ba3uX9FL8MYu7+MvAMsGzdsUhrc/ePKT+Li9Ydi4gMHhGxLHAL8F3gYWBNd/+FqpRE+pe7j3P3U4AvAscAcwAXRMSYiFi93uhkIFNiSVrFmcDrZnYbJaE0MjMfq54bB5CZr1GGt+1bLQ/gjcy8tXr+Gqq+I8AY4MLMvKHf9kAGuycofZY0a4fU7X5g4Yj4bN2BiMjAVlUp/YRyc25eSo+Xjd39hXojE2lt7v6Ru/+EMnnMKZReZ1dFxNVVIlhkslimbhSIdMXMtgZWzMyRdccig19EbA+cA2zl7lfXHY+0rohYiVJZcKi7/6LueERkYKouTs+gVOPeBuymhJJIc4qI2YHjgc0ofWdvAQ5096frjEsGDs1AJNIFM7sMmB3Ypu5YpGVcA7yRrNAGAAAgAElEQVQEOKDEktTpAUqfpQ0AJZZEZLJExBDgOErfyveBHwCnaNibSPNy97eAvSNiHuAEYENK/6W/At9z95dqDVCaniqWRESaRETcDJi7r1V3LNLaIuIqYAF3X7HuWERk4IiI5SkNgZdmfJXSi/VGJSKTKyIWoswWuy7wHmUG40Pc/Y1aA5OmpR5LIiLN4zFgaETMUXcg0vLuofT8UhNvEZmkiJg2In5Oqb79FHAwsImSSiIDk7v/0923pySW/h+wA3B3RPw8ImauNzppRkosiYg0jwDmATTtq9TtPMpw+e3rDkREmltErECpTtqL0vx/TXf/jYa+iQx87v6wu28GfJkyTH4vSoLp8IiYvt7opJkosSQi0jxuBJ4Dtq47EGl5D1NOINepOxARaU5VldIJlL6AcwNtwGbu/u96IxOR3ubuY919fWA74A3gEEqCab+qr5q0OPVYEhFpIlWTxFndfdW6Y5HWFhFXAAsDy6vyQEQaRcSKwGnAUsDNwAg19xVpDRFhwFeB71P6qT1K6cd0js4XWpcqlkREmstDlN4289UdiLS8OyiJpcXrDkREmkNETBcRJzK+Sukgd99MSSWR1uHu6e4BfAk4CBgCjAJuj4jNag1OaqPEkohIczkHmB3Ypu5ApOVdUP27Q61RiEhTiIiVKb2U9qQ0+F/d3U+rNyoRqYu7j3P3U4FhwI+BWYELIuLGiFiz3uikv01bdwAiIjKBu4FnKE0ST605Fmltj1N+FlevOxARqU9ETAf8DNgJeBs4wN3PrDcqEWkW7v4RcHxEnAwcSZn444qIuBNoc/cHaw1Q+oV6LImINJmIuBqYz91XqjsWaW0RcRnwOWBZ9U0QaT0RsQrlJseSlAkmRrj7K/VGJSLNLCJmB44DNqdUMd0CHOjuT9UamPQpDYUTEWk+dwNDI2KxugORlnczpc/S0nUHIiL9p+ql9H/AlcAcwHfdfXMllURkUtz9LXffF/gicAVlqNxtEXGOeogOXkosiYg0n3MpQ5W97kCk5V0IfEwZAiMiLSAiVgVuB74BjAXWcPez6o1KRAYad3/F3XejNPm+BdgIGBsRoyJiznqjk96moXAiIk2mmsb1b8Az7r553fFIa4uI+4EX3X2TumMRkb4TEdMDJ1BuarwBHOfuZ9cblYgMFhGxBHAisAbwKnApcIS7v1drYNIrlFgSEWlCEfFnYFHU20ZqFhGXAEsAy7j7uLrjEZHeFxFfAn4NfAEYA+zu7q/WGpSIDEoRMQw4njJE7nngPOB4d/+w1sBkqmgonIhIcxoDLAIsW3McIn+l9Flaoe5ARKR3RcT0EfEr4PJ//etfq4wePfr5HXbYYc4ddtjhBjP7kZlN8lrBzA6c3Pc1sxXN7JfV98ub2QZTEL6IDEDufpe7bwBsC7wG/AC4OyL2j4gh9UYnU0qJJRGR5nQh8B/ga3UHIi3vj8D7qM+SyKASEatTeintCtx2+OGHc9VVVy2XmesAKwOLAUf0YFP7T+57Z+b9mblf9XBlYO3J3YaIDGzufj2wFrAb5Zz3OEoPpl2rthAygCixJCLShNz9GeAZygm3SG3c/TnKz+KKdcciIlMvImaIiFOAy4BZgL3dfet33333v+tk5sfAd4H9zGxmADM7wszGmNmdZna0mc1kZmOA+avlS5jZSDM73MxuNLPdzGxmMzujenynmW1SbWto9ZptgEOAEWamBuEiLcbd090vojT4PgAYAvwGuD0i1Gd0AFFiSUSkeT0NDI2IaesORFre08Ai+lkUGdgiYg1KldIulFmaVnf3P3S1bma+BvwTWMLMdgHmyczhwGrAcsC61eMXM3N4Zj5WvXQnYNvMPBv4IfBSZq4LbAOcZmbzNLzHnyi9VkZn5u69vsMiMiC4+zh3Px34InA0MBtwXkTcFBGqaBwAdIIoItK8rgA2o9zFubXmWKS1XUP5WRxGuSgVkQEkImYATga2p8zGtJe7n9uDl04LjAO2BJasKpQAZgeWBK7u4jXXZWZH4+9NgG8CZOa/zOweytCX+6ZwV0RkEHP3j4GfRcQvgMOBrwKXR8RY4Pvu/kCtAcpEqWJJRKR5XUaZ8nnnugORlncp8BawQ92BiMjkiYg1gTsoPftuplQpTTKpZGZzAwsAj1OGpxxTVSYNz8yVM/Pkibz0nc6b6vT4k8naARFpOe7+gbsfTqlg+h2wOPCXiLg0IharNzrpihJLIiJNyt1fpvS2WabuWKS1ufu/gWcpw19EZACoeimNoiSGZwK+5e7buvvrk3qtmU1LqXAalZnvUSqTdjKzIdXzo81srWr16TqWd+FaYLvqNfMBy1OG4DV6D5h78vZORFqBu7/t7t+lJJguB1YBbo2IcyNi/nqjk0ZKLImINLcnKL1tZqo7EGl5T1F+FqevOxAR6V7Vk+QOYEdgDLCau5/fg5f+xcxuBh6lTAN+VLX8TOAx4DYzuxV4LDM7EkRnAfea2V5dbO9YYNFqmxcC36h6NzW6FhhmZrea2aI93kkRaRnu/qq7705pD3ELsAFlBrnTI2KueqMTAMvMumMQEZGJiIjtgXOArd39qrrjkdYVEbsCpwEbu/tNdccjIv8rImYEfgFsC7wMHOnuF9YblYhI74qILwAnAmtQkuCXAUe4+7vdvlD6jCqWRESa2zXAS4DXHYi0vCuA1ykVECLSZCJiXUqV0g6Mr1JSUklEBh13/7u7bw5sSGkb8U3gnogYqcrqeqhiSUSkyUXEzcA07r5m3bFIa4uIO4AP3H143bGISFFVKf2SUqX0b0qV0kX1RiUi0n8iYn3gR8CKlKH7ZwK/dHdNFtBPVLEkItL8HgOGRsScdQciLe9JYOHqQlZEahYRwylVSg78hTLjm5JKItJS3P2vwNrAbsD7lP5ud0XEiIjoPDOl9AEllkREmt/5wKeBzeoORFrepcCClJM3EalJRMwUEWcCFwPTAd/w4s2aQxMRqYW7p7tfTGnwvT9gwK+BOyJiq1qDawFKLImINL+bgeeALesORFre1cArwFfrDkSkVUXEBpQqpe2A6ylVShfXG5WISHOoEkxnAMMoM1vOAvwhIm6KiHXqjW7wUo8lEZEBICL+Cszm7sPqjkVaW0TcBnzs7jo5E+lHETEz5e77lsCLwA/d/ZJ6o5JWYmYJ3ESpBJkTuAQYmZnjzGwTYJXMPLbOGEU6i4gZgCMoN8XmB8YC33f3+2sNbJBRYklEZACIiJOBnYHl3f3FuuOR1hURvwPWBZZy9/fqjkekFUTEhpSptRcGrgK+5e5v1xuVtJoqsTRdZn5sZtMCo4HHM/PoeiMTmbSImJXSe2kLYA7gVuBAd3+i1sAGCQ2FExEZGM4FZqPM+iNSpz9S7vitX3cgIoNdRMwcEaOBoJy37+ruOympJHXLzI+B7wL7mdnMZjbczEYDmNloMzvAzK4zs7+Z2U7V8vnM7DIzu8HM7jGzkzq2Z2ZPm9kPzOzWWnZIBj13f8fd96cMkbsUWBm4NSLOi4gF6o1u4FNiSURkYLgHeAbYqO5ApOVdT5nSfPu6AxEZzCJiY0ovpa0p/c1Wd/fL6o1KZLzMfA34J7BEF0+vBGxMqXA9rlq2NPDrzFwP+CKwkpktVj03L/BRZq7Zt1FLq3P3V939G8CqwI2UG2V3RsQZETF3vdENXBoKJyIyQETE1ZRKkZXcXQdvqU1E3AyYu69Vdywig01EzAL8Btgc+BdwiLv/ud6oRCYcCtew7EFgF2AuYERmjqgqlyIzr6zWeTIzFzOzGSnNlNcGPgGWBLbPzJurbc+VmW/0715Jq4uIxSlDjdcEXgcuA45w93dqDWyAUcWSiMjAcTewCLBo3YFIy3scWCQiZqs7EJHBJCI2AW6n9AC5Clitu6RSRCwcEddFxOf7K0aRDmY2N7AA5W9CZ409+DquOX8MzAdslJnrArcAQxrW04W89Dt3f9zdtwA2AJ4C9gDujoijq8bf0gPT1h2AiIj02DmUfgY7Mr6sXKQOFwJfowxz0DTnIlOpqlI6lVKl9Bywi7tf0c36BhwC7EU5n18Z+Ec/hCoCQNW8+2RgVGa+Z2Y9ednngEsy830zWwIYDvyy76IU6Tl3vw/YMCLWA34EHARsGxFnAf/n7p/UGmCTU8WSiMjA8Silz5L6D0jdxgAvUHq/iMhUiIivUHopbQ78mVKl1F1SaTHK7+APgaeBddw9+j5SEQD+YmY3U85JXqMMbeupHwP7m9mNwA+ASyhD6ESahrvfAKxDGeL5PnAMpYJp9yqpL11QjyURkQEkIi6j3PFbzt3H1R2PtK6IuBGY3t1XrzsWkYGomvr6VOArlAbIP3D3q7pZfxrgcGBPYFz12uPVc09EpG9UiaTdgf2ALwAPAzu4+5O1BtaEVLEkIjKwjKH0WVqu5jhEHgUWjgjdbRaZTBGxOaVKaTPgcsqMb90llZYAbgIOpgx5W8fdf6KkkohI33H3dPffAsOAIyn5EzWY74J6LImIDCwXUYY/7Aw8UHMs0toCGAFsCpxbbygiA0NVpXQa5ffmn8BO7n5NN+sPofT6GAF8SBl2dKISStIKImJmygX9XA1fcwAvuPupdcYmrcXdPwZOqL6kC0osiYgMIO7+bEQ8Q2nUKlKnWyhNhrdEiSWRSYqILYCfAJ+hTGf9HXd/r5v1lwVOp1So3gHs5u7P90esIk1iRsrEJQt2Wn42ZSioiDQJDYUTERl4nqRM9a6bA1Ibd/8PpZn8onXHItLMImK2iDgP+D2lN9IO7r7bxJJKETFtRPwMuBaYFzgU2EhJJWk17v4asA2luq/DO8DDVTWfiDQJJZZERAaeq4CFgdXqDkRa3sOUJOc8dQci0owiYitKtdGXgT9RZny7rpv1VwRuA/YG/gas6e6/1NA3aVXufhdwNPB6teh1YCRwT0SMjIhZ6opNRMZTYklEZOC5lHJitXPdgUjLOw+YE9ii7kBEmklEzB4R51OG7HwMfNXdd++mSmm6iDgJuJrSR+YgYFN3f7HfghZpUu5+JmVI3BvA9pQZut4D2oD7IuL0iFigxhBFWp6GUYiIDDDu/krVZ2npumORljeWMkRhM+C3Ncci0hQiYhvgWGB+4GJgX3d/v5v1VwVOAZYAbgR2d/eX+yNWkQFkf+Bxdx9L+dtzZkSsBxxCGS73lYgYC4x09/trjFOkJVmmKmtFRAaaiDgXWB1YursLFpG+FhHXA3O6+xfrjkWkThExO6XZ9ibA08CB7v7XbtafHjgRcEoV6k/cfXTfRyoyuETE4sCPKS0C5qDMmnsScJmGkYr0Dw2FExEZmC6mzCy0ft2BSMt7EFhYwxCklUXE9pReShsCF1J6KXWXVFqzWn9XSk+l1ZVUEpky7v64u+9ImTH3bErT+3OBsRFxQJXEFZE+pKFwIiID0zXAv4GvAlfUHIu0tnOBb1OGIpxScywi/Soi5gDOADYGngK2dfcbu1l/RuD/gG2BV4G93P3c/ohVZLCrZpE7ICJmAPYFdgKOA/aMiKuBY9z99e62IdJfzOxYYC9gvsz8uA/fZ2Hgi5n5x+rxycDvM/Oe3nwfVSyJiAxA7v4OZajF4jWHInIv8CywUd2BiPSniHBK1dH6QFCqlLpLKg2v1t8RGFOtr6SSSC9z9/+4+wnAMOBrwMvAHpRG3+dGxGK1BihSdAybXm9SK5rZimZ2s5lNSfXdYsCWHQ8y84DeTiqBeiyJiAxYEXE6sCmwrLu/UXc80roi4hpgHndfue5YRPpaRMxJqVLaCHgS+K6739zN+jMBvwa2olSaHuHuF/dHrCJSRMQqwJGUZNN0wD3Ace5+U62BSUsys2UofcHuBBbNzL3MbAzwB0oV+DeABYEfAR8BiwDLZebrZrZG9drpgLeAb2fm89Xrz6P07fsU8D3gfeA3lMkk7svMTc1sNDA6M8eY2f2UkQ9rAUOAHTPzOTPbEvg+MA6YDfh+Zv6lu31SxZKIyMB1ATAPZUYukTrdCywSEQvXHYhIX4qIHYHbgeHA+ZTeSN0llTamXDhsC1xLqVJSUkmkn7n7Pe6+FSWxdDnwOeCKiLgtInaLiCH1RigtZjvgsuprGzPryMusB6yWmQ8Co4GvZeaalITTlmY2J/BLYLvMXIcyYcQJDdtdMDM3oAwDPTYzbwcOAK7OzE27iGMZ4KbMXBc4B9inWr40sEW1fHvgqEntkHosiYgMXDcDz1HKWzWcQup0PvBdyonSSTXHItLrImIu4ExgA0qV0tbufks3688CnApsTjlOf93dL++PWEVk4tz9eeAbETEr8APKOdRvgAMj4hLg5+7+bp0xSkvYGtg4M18xs7eAtavlF2bmJ9X3L1NmOaTh3zWAhYBLzAxKoVBjsdD5AJn5iJnN24M4Xs/Ma6rvHwE6Zvi9EjjNzBYEPqZUT3VLFUsiIgOUu/8HeIZy102kTg9SfhaH1xyHSK+LiJ0oVUprU5L4q00iqbQ5pUppc+DPlKomJZVEmoi7v+PuR1BmktsPeA9oo/RhOl0znUpfMbPPAwsDfzCzqynFPttXT7/TsOrFwKXVELc3gd9ThqvdnZnDq691MnOthte81/B9T6rw3uv0eIiZDQGuBn5bVUtt3pNtqWJJRGRgewDYNSLmdfeX6g5GWpO7j4uIZ4ChEWHurgaOMuBVVUpnUZpz/wP4hrvf1s36swGnUXrfPQPs6O7XTGx9Eamfu39CqUY8MyLWAw6hzHL6lYgYC4x09/vrjFEGne2AtswcDWBmqwJ/BJ7oWKEaGrcrsFBmjmtYfjtwipkNzcynzWxTYLPM3K+b93sPmHsy4pu1+uroP/btnrxIiSURkYHtp8CNwCt1ByItbyywP2X2kScmsa5IU4uInSk9Jeam9LY4sKoSndj621Gaqc5HuUDY19073wkWkSbm7jcAN0TE4pTf59Wqxw9Qhnlfphsn0gu2oFQBAZCZY83sDWD1hmXjzOw54E4ze5fShPuvQDuwO3C+mX0AvAF8axLvdx8wvZndQem31K3MfNPMjgHuMLOXgd8BM0zqdZoVTkRERKZaRCxFSS4d6+7H1x2PyJSIiLmB31KqlB6nJIhu72b9zjPEHejuY/ohVBHpY9Xx4EhgY8qsXA9TGhyf4u4f1hmbDG5mthHwjczcqXo8K3ADsGdmNmUFnRJLIiIiMtUiwii9lp5y9y3qjkdkckXE14HDgbmAAL7X3cVj1XvpSMrsnBcD+7v7B/0Rq4j0n4iYAdiXMtPW0sBTlB40x7j763XGJoOTmX2KcpPj05Tm2TMCdwEHZObHdcY2MUosiYiISK+IiD9T7uour+ECMlBExKcovZSGU6qU9nb3O7tZv7Gq6R/Afu5+az+EKiI1qm6gbE0ZTrQCZRjSbcDh7v5knbGJ1E2JJREREekVEXEYpfHpqu7+aN3xiExKROwGHEapUjofaJtElVLj+pOsahKRwSkiVqFULA4DpgPuAY5z95u6faHIIKXEkoiIiPSKiPg8cDdwsruPrDkckYmKiE9TqpTWBf5OqVIa28368wCjgXWAxya1voi0hohYEPgR5dgwP2VI+CjgD9WMcyItQYklERER6RXVMIH7gefdfbO64xHpSkR8g1JZNwfjq5Q+msi6Rplxpw2YDTgPOHhi64tIa4qIWYGDga2AxSkJ60uAn7v7u3XGJtIflFgSERGRXhMRlwGfB5Z193F1xyPSoapSGk2pUnoU+I67393N+vNX669JmQ3qW+7+QN9HKiIDVUQMAUYAewDLA/8CxgBHufsL9UUm0reUWBIREZFeExHfowwLWN3dH6w7HhGAiNgD+AEwO5OoOqqqlPYD9gdmAv4AHObuTTkTj4g0p4gYDhwKrAJ8CIwFRrp7U04XLzI1lFgSERGRXhMRiwD3AaPc/dC645HW1qk30qPAt9393m7WXxA4G1iN0itlT3d/qB9CFZFBKiIWB35MOa7MATwAnARcphlUZbBQYklERER6VUTcB7zk7l+uOxZpXRGxJ6XnySR7I1VVSt8D9gWmpzT2PkrNd0Wkt0TE3JSZ5DYGFqEMsT0HOEWzS8pAp8SSiIiI9KqI+BOwFLCMLsylv0XEvJSqo7WARyi9kSY69KSqshtNmTb8fmAPd3+sH0IVkRYUETNQktg7AUsDTwFXA8e4++t1xiYypZRYEhEZRMzsUkqZ9ZLAm8ALwC2ZeXgfvucmwCqZeewUvHY0MDozx/R2XFKfiNgH+DmwtrvfU3c80hqqqqNvAwdRqpT+ABwysd5I1fqHVq+ZBjgD+LGazotIf6iOQVsDBwArAG8AtwGHu/uTdcYmMrmUWBIRGYSqhM0tmXnGZLzmMGD6zBzZSzF8lnJht2tmPttNnEosDTIR8RlKD4nfuftBdccjg19EzEepUlqTUqW0Z3czuEXE5yjD3VYG7gV2d/cn+iNWEZHOImIVyjC5YcB0wD3Ace5+U62BifSQEksiIoNQ58SSmR0BbECZ4ejqzDzKzAw4gTJcZAjwaGZ+zcxGAEM7EkyNyR8z2ww4ijK7yV2Z+T0zGw6MyMwRZvZF4BfAJ8BCwNaZeX/1/l8GrHpu58x8rtO276DcqRtWrfP1zPynmW0JfB8YR6lC+H5m/qXPPjzpFRFxN/C6u29UdywyeDVUKbUBs1CS2Yd2U6U0DXAE8E3KMeVU4Hg10BWRZlBNIPAjyoQD81MmERgF/EFDy6WZTVt3ACIi0rfMbBdgnswcXiWTLq6GrwGsCnwJmAG418xm6mY7n6Kc3AzLzBfN7HQz2xx4p2G1XwFHZOZ1ZrYXsJWZPUFJFK2dmWlm3wC+Rbkz12hl4KAqWfVN4JeUEvGlgS0y8w0zW4zSC0WJpeb3NLB8REyradqlL0TE/JQqpTUoTXC3m0SV0pLAmZQhJ2OBEe7eZTWliEgd3P15YI+ImJUy+cBWwG+AAyPiEuDn7v5unTGKdEWJJRGRwW9LYEkzG1M9np3Sg+kWSkJpemDm6t/urAHcm5kvVo8PAP5DqXjq8DKlxxPAnACZ+baZPQJcZWazVO9/dxfbfyEzb62+D+DE6vsrgdPMbEHgY2DBScQpzeEqYHNK8vK2mmORQaSqUtqbMovbTMCvKT1JJlalNIRSATCCcsw6EjhJVUoi0qzc/R3gyIg4mnLs2oNSmblzRIyhzFr5Qn0RikxIiSURkcFvCHBMZkbjwirJMwPlzv1LwD6Z+b6ZjaMMWeswY/XvNJ2WW7Ws0f9n777j5KyqP45/DimEEiBBOpggHaRXqUGQANIieEHFH6EJigLqUKQGEEEcRBHpJXS4iCI1KEhogtQQmgJC6EgntEBIzu+Pc5cMw+4mbHZ3tnzfr9e8svPUOwPzlPOce+6VwGlmti+RQfAzM1sJOAnY1N2fMrOtge2baafVvXcz60OMlLKbu48xs4FEWrh0fdcAxxKj3iiwJO0i57wQ07KUHiFqKbV4TMg5fxU4C1gRuAvYJaX0Ume0VURkZpXub+cA5+SchxEDDnwL2CrnfA8wqrVRL0U6iwJLIiI93xjgO2Z2pbtPKXWNziZqIN3g7gfVLf8csDVA6Xq2MVGH5C7gVDObz91fK9MuAibVrHsgsLi7f9o9zsyGAk+XoFI/YFdgYjPtXNTM1itZS9sBY4E5y6upeOVebfoGpNOllF7NOT8HrNDotkj3V7KUfkxkSs5GdJU9rKWaIznnvsBxwM7AB8DBwB+VpSQi3VVKaSwwNue8FHAMsA5wS855HPA74God46RRFFgSEen5zgGWAP5pZp8A17r7HWa2NHCyma0LTAbeJDKLbgV2N7N/Ai8BfwZw91dLJtJ1ZjYZuL1kEQ2r2dd9ZT9vEQGne4FfA98zs3uIoXT/DGzRTDufB5KZHU/UZNrF3d8xs18Cd5vZa8AFRJaVdA/PAGvknPunlD5udGOkeypZShcAXyMyFvdIKT3ayvKrAGcCywF3ErWUXmlpeRGR7iSl9CSwU855XmIwgs2AS4DHcs4XA6fqnCudTaPCiYj0UmZ2MXCBu99YinqvCZzp7qu0cXtLA+cyrUj3AOBS4Hx3v2oG1p/g7kPbsm/pmnLO3yP+nxhenrSKzLCSpbQvsB/RJfc84IhWspT6Ab8BdgLeBU5IKZ3VSc0VEWmInPOsREbnd4gBT54BbgSOSSm91ci2Se+hwJKISC9lZlsARxKZRX2JGkcnuvuf27i9AUSWwFJEgdwBwFPAj9y9ua5v9esrsNTD5JwHAY8S6fl7N7o90n3knBcmspTWAcYDu6eUHm9l+bWAU4FliG60u6WUXuuEpoqIdAklGL8d0WV4ZSJL/J9Et+GnG9k26fkUWBIREZEOk3O+C/gopTSs0W2Rrq/cGO1PZCr1J7KUjmwlS6k/0YX328BbwK9SSud3UnNFRLqknPPqRDe5tYB+wP3E8fG2VlcUaSMFlkRERKTDlHoP6wLLp5Q+bHR7pOvKOS9CZCmtDTxEZCn9u5Xl1yOKeC8J/IPIUnqzM9oqItIdlOPq0cCGwIJEnbozgItaCtiLtIUCSyIiItJhcs47ABcD26SUbmx0e6TrKVlKPyNqhPQj6nKNSilNbWH5AURAaQTwGnB0SunSTmquiEi3k3Oekxi5dxtgaeAJ4CrgNyml9xvZNukZFFgSERGRDpNzngt4DLgxpbR7o9sjXUvOeTHgfGLwgPFE1tF/Wln+68BvgcWBvwF7ppTe7oy2ioh0dznnPsAuwB7ASsTov2OJLscvN7Bp0s0psCQiIiIdKud8JzAlpbRho9siXUPJUvo5kaXUFzgHOKqVLKXZgT8ST9tfIUaHu7KTmisi0qOUY/Aw4GBgdeBj4B4iW3RcA5sm3ZQCSyIiItKhcs7nAxsDyynlXnLOX2ZaltKDRJbSk60svzlwArAYcAOwV0rp3c5oq4hIT5dzXgo4hhiFc25gHPA7YkRXBQtkhiiwJCIiIh0q57w18Cfg2ymlqxvdHmmM8oT8AGAfYBbgLAnWDDoAACAASURBVOCXrWQpzQmcDnwTeAH4RUrp2k5qrohIr5JznpcYSW4zYAjRjf1i4NSU0seNbJt0fQos9SBmNgQ4BVgIcODP7n5cB+5vFDDB3Ud31D5ERKT7yznPQVygjk0p7dLo9kjnyzkPIbKU1iCylHZNKT3VyvJbA8cBiwDXAD9UtpuISMfLOc9KdFP+DrA88AxwI3BMSumtRrZNui4FlnoIM5sFeAA42d3PNbP+RLr42e7eISOlKLAkIiIzKud8G9AnpbReo9sinadkKR0E/BAwIkvp2FaylOYCzgQ2B54Dfp5S+nsnNVdERIpy/N4O2B9YGXgb+CdwWErp6Ua2TboeBZZ6CDMbDvzB3ZeumbYasBywBVGXYAARcf6uu081s3HEkL4jgIHAru7+sJl9DTgemAoMAk5w90vKNo9h2sXeu8BYdx9tZocDw4mLxillHy90wkcXEZFuIOd8NnH+WCGl9E6j2yMdL+c8FBhNZCk9QGQp/beV5XcAjgYWIIbB/klK6YOOb6mIiLQm57w60U1uLaAfcB9wXErptoY2TLqMvo1ugLSb5YBHaie4+wNmNgWY2pS1ZGYXECMA/IMIJn3o7hub2RbAIUTK40rASHd/xszmAu4GLjGzzcu6XyNqI9wKjDWzgUQwaQN3dzPbDfgBcEQHf2YREek+MvB9onbDFQ1ui7SznPPApoLa5Sn3wcDexAOn44gbkJaylOYhRoXbFHga2D6lNLYz2i0iItOXUrof2C7nvAjxAGBD4Lqc88PAGcBFKaUpjWyjNJYCSz3HJ0SGUb3xwBZmdhtRd2ko0JRS3ocoyAbwODB/+XsMcJSZLU0MPTm0TP8GUbfpEwAzuwnA3d81s8eBG/r16zf3PPPMM3SppZbynPORGklARESK24CXiLR6BZZ6kJzzbMBDOeddiCLb5wOrAfcTWUotdpnIOX+XeBA1L3ApsH9KaVLHt1pERL6olNKLwO5lcIUDgW2A04Cf5pyvAn6jeni9kwJLPcd44Ce1E8xsbWLYyI2Abdz9bTM7kQgoAeDutSnmTdP/RNQ32K10mXujTJ+lbp8GsMoqqwybffbZzx01atR/hgwZssj999+/4K233vqmgkoiItIkpTQp5/wssESj2yLt7nhgcaJ+0uzEg6xjgeNbuhbIOQ8GzgM2Bp4Cdksp/bNzmisiIjMjpfQecETO+Wjg/4A9gZ8D3805jwWOTCm93MAmSidTYKmHcPfbzGyime3u7ueY2azAr4gLvdElqLQA8aT44elsbgngxhJU+g7RZQ7gFmA/M/s9Ua9pS+DklVZaad+PP/54ziFDhqz9ySefcMstt9CvX7/+OefrgdfK/v4F/Bt4XQEnEZFe63Fg25zz4JTSm41ujMy80i1iu/J2GeJcv0VKaUIr6+wK/AKYG7gA+JmGshYR6X5SSp8A5+aczyNKphxM1O/9Zs75XmBUSmlcA5sonUSBpZ5le+AMM/shUfPoEqL45aVmtinwP6L7waDpbGc/4Doze43oFvekmQ1y96vNbH2i5tLbwB0AF1544Y7zzTffzQcccMAa88wzz2xrrbUW99xzD0TXuuWIC87ZgbeAN3LObwJvAK8SxTzvAZ5UMVcRkR7vcmA34JvAhQ1ui7SP04Av17wfDCwKTKhfMOc8H9FNbgPgP8D3Ukr3dkIbRUSkA5XEgVuAW3LOSxF1mL5W3o8DfgdcrQSDnkujwkm7yTnPTVwwbgKcmlI6KOfcDxgCrEh0y/sKcdE5b/l3MDGyQFOwqenfl4mA0wPAUxoVRkSk+8s59weOBE5LKWnk0G4u57wOcA3wTnm9Xf79U0rpkprljBjUowLMSTz4OjilNLnTGy0iIp0i5zwvMZLcZsT94GNEfd9TlaXa8yiwJO0u57wH8HBK6V/TWW4A0e1uFWLoysWIgNMgpgWenM8HnZ4jsqbGA0/rwCQiItL5cs5zEYGiV1oZ8W1B4qHTusRNxQ9SSg91XitFRKSRcs6zAj8mRh9fjshovRE4JqX0VgObJu1IgSXpknLOA4EliVFl1gQW4vOZTh/x2aDTm8CTRNDpUeBZDXspIiLS+UqW0k+I7vWzEV0fDy31OEREpJcp54XtgP2BlYks138Ch7U2eqh0DwosSbdTRpJZmshyWoWo5VQbdJoHeJ/PZzo9RgSdHgdeUh9fERGR9lcKep9PdIF/GNgjpfRoY1slIiJdRc55daKb3FpEWZT7gONSSrc1tGHSZgosSY9RouALECmWawFfBb7EZ7OcBgIT+WzA6XU0cp2IiMhMKefhCvAj4kZhNDHktLKHRUTkc8qDiKOBDYEFiXuyM4CLdO7oXhRYkl4h5zwLMUrNCsDaxJDI8/LZoFPTyHW1mU7/QyPXiYiItCrnPITIUloDGAfsllJ6orGtEhGR7iDnPCdwILAN0TPlCWJ089+klN5vZNtkxiiwJL1eGbluKJHh9DVgcT5fRLxp5LraoNNLRJbTg2jkOhER6YXKg5uDgb0AA84Cjm2pmLeIiEhLcs59gf8D9iRGFf9TSmlkQxslM0SBJZFWtDJyXVOW07zAVD4fdHoWjVwnIiI9WM55SeBcYqCN+4kspf82tlUiItLdla7VGxE9Rl5sdHtk+hRYEmmjmpHr1gBWZ8ZGrnsDeIoIOj0CPKf+wyIi0tXlnPs0na9KltIRwO7Ew5VTgRNUn1BERKR3UmBJpAPknOcl+gevybSR62oDThq5TkREuoWc86LALURm0iJEltJKRP3BkSml5xrYPBEREWkwBZZEOlHdyHVrM23kusE1r7mAd/hs0Ok1olvd3UQxO41cJyLSw1TNLgJuqbifU96fDEysuB9W3v+I0gW74n5sZ7Ur53w1sDUw7pM331zwrbPOGjjfQQcdAZykc5GIiIgosCTSRZSuBYsByxNFxJfm80XEWxq57n7gXjRynYhIt1U12x3YsOK+S3k/Hvi44r5GeX8xcHLF/V+d1aac8yZAJs5BEOea7VJKL3VWG0RERKRrU2BJpBtoZeS62iLifYG3geNTSic1pqUiItJWVbMlgJsr7kOrZgsB5wErAKtV3F+rmj0K7A98r+I+smp2APB9op7ff4HdK+7vV822BI4EPgburbj/rGo2C3AcsC4wK/DHivv5VbOhwCnAC8Q55j0gAe9idmLfBRfcywYMmH3WpZZi0K67YrPM8vBzO+00d2Xq1CFVs5GlfUsDiwJjK+4/74zvSkRERLoOBZZEurmaketWJbrXnZZSeqyxrRIRkbaomj0HbACsDywMLEvUN7oFOB/4JTCSCDC9CAyquH9cNduDqHn0IjAOWLPi/krV7Czgr8QAE8Mq7t+rmg0E7gO+RdT7ewxYq+L+SNXs18DTwL9mmWOOmxY999x+wMS3r7ii/xwbbPB8vwUXfOe5nXdevvLxxwuVwNIBwDpEQOouYJeK+386+nsSERGRrqNvoxsgIjMnpTQJeLS8Lmpwc0REZOaMJYZY3hj4AzAB2BaYDNxes9xEovbejVWzm4DbK+7jq2ZbAw9U3F8py+1PZDRdTjlHVNzfrZpdD2wG/AV4pOL+SFn+caJb9tNTJ0167/nddnuaKVNu9kmTbvjBFVc8AFDdcccJNe24ruL+LkDV7N/EYBUKLImIiPQiszS6ASIiIiLyqVuIwNIqRObR34GvE1lMtzUtVHGfSmQ1HULU3juxanYEcW1nNdszpl3v1U4HmFL+/aBuep+K+0SmTFna33//aJ806QPg0qrZLs2093PrTv8jioiISE+iwJKIfIaZLWpm15jZPWY2zsz2LdM3NLPVapb7aeNaKSI9iYVflePO7WY2xsyGNLpdDTIW2B74b8V9asX9beAZou7R3U0LVeP7OR24p+J+KvBbYGWiO9rqVbP5yqKnE4GpvxFd36iazUFkK93YUiOqZmsCR1Xcx1bcTwJGAyu138cUERHpfGZ2rJm9YWZ9y/uxFvUGZ3a7C5rZFTO7ne5KgSURqXclcLa7r0UUCh9hZiOIG5Pam4r9GtE4EemRdiTqxK3j7hsAvwEubmyTGqPi/gwxEMPNNZNvAJ6quH9Ys9yzRC2ku6tmdwB7A0dU3F8F9gWuq5rdCbxQcR8DnAO8VTX7JxFQGjWdWkgPAAOqZveUdTYCTmy3DyoiItIYmxPdzDdubSEzO8TMRs3oRt39FXf/9sw1rftS8W4R+ZSZDQOOKTd2TdPWBa4D3gEmEfU4lgeGE0/PjwReBk4lukAMAi509xPN7AIgu/u1ZnYoMIu7H2NmSwM/BSrAGUQ9jwHEU/nvenTxEJFewsz2BzYFtnX3KWXabETAaai7jyrTRhOZM9sCL7j7iWX6WcSxZGGimPRUYCBwgLvXBmhERESklzKzFYBjgH8Bi7v73mY2lhgU41niAcr6xD3Nv939e2W9w4FNgNmAMe5+ZMlyGg08BCzi7juY2QR3H1rudT53b9RZn7MRlLEkIrVWBB6sm/YwUeh/NHC8u//C3bcFXnH3Ye5+K5HJdLC7bwysCexhZrMC1wJblO2sRwSjALYinsAvCVzn7hu5+9rE0NjDOurDiUiXdSExXP3zZnaRNV/Lp9YlRNcwSir7Su5+HxH03trdNwJ2IALf3YqZuZldUzetr5m9Wi5+MbPNS7C+te1MaGH6wWb2zfZqr4iISDeyPXB1eY0ws9p4yHBgLWKU7fWAVc1sNjPbGZjP3YcRo6CuaGabl3VWB+5w9x3q9tPSvVGPpcCSiNRqruhqH2I0otb8A9jVotvF34msgfmJ7habmtmcxKhEH5vZ/MA3yjrjgSFmdpuZ3Up0tVikXT6JiHQb7v4G0RXuW8BjRDbj3UCzF2Hufi8wj5ktTqSy/6PMuh44sxyLzqN7Hk9eAtYwswVqpm1BZIYC4O5j3P3Ytmzc3Y939+vqp5vZ8PqAloiISA+zHXCtuz9OjK66Qc2814nrjv7A7OVfgG2AYeXhzi3AUGDZMm+KuzdXV6mle6Meq2+jGyAiXcojRCS/1mp8Poup3hnEENXD3H2ymd0P9HH3d8zsReCHREHavsB3iW6475nZPkQwaRt3f9vMTkQjCon0OhaFoh9297uBu83s10SAaQE+O5LZgJq/LyOylpYE/mhmfYAxwG7uPsbMBhIZl93NZOJJ6veIgtwA3wfOJy5um7otj3T3kSVYfzYwD9FdeU+P+ktYjBK3GTH623fd/cVSL2KCu482s/8j6jFNIboOrmhR2LvZLspmNg44FxhRlt/V3bvjdywiIr2MmS0JfBm4yMwg7ktqM40eJwJL9wCvAvu4+4fl+uKX7p7rtjcUeK+F3TV7b9R+n6brUcaSiNS6GZjbzLaDT2ucHAUcTwwpPbhm2Q/MrOn9EsBN5cC5AdEdpcm1wKHEiETXAkcAN9Wsd1cJKi1APEUQkd5nW+DYmpT0eYmgxmOU44mZfYXPFtq8hAhUL+/u44A5y+u2Mn+vTmh3Rzkf2AXAzOYBFieKaTfnd8AN7r4hcbxuquGwCPCgu68PXEoEkD5VMklPBjYtXZHHE3UlWuuiPBD4sKT2Hw4cMvMfVUREpFNsD1TcfXN335yo4ziCaQ+wtiHOpyu7+zfcvWnk1DHAd0qACTMbbWbrT2dfrd0b9UjKWBKRT7m7lxHgTjezw4jI+unu/ncz+y/w5xJ02oy4IbnDzDJwMHCKmb0J3E+kiQ4iRly4Fti/pJxiZq8TB2iA3wOXmtmmwP+AK8p6ItK7HAGMAu40s0nEsWcvojvtthajkr0E/LlpBXf/j5l9QukGVzIkf0lkPL0GXEALXem6One/38KqRG2Gy1tZfDiwT1nvTjN7pEx/x92burb9h6gZUesj4H1gTjN7lwgaQQSYtjCz2wAnUv7/Xub1YdpofY/Tw9P6RUSkR9maqPMKgLvfY2ZvE6NgQ9zDnFwGLpoMvAmcRIyqugTwz3Ldca2731EyllrS2r1Rj6RR4URERES6iJoRZX5OpOyvRGRmLQOMcvdhdV3h3gKWcPc3y/qD3f3Npu2UabXLj2JaV7jRwIbAc8DV7v7b0kV5K+A7NV2UHy7L125zKDC6FDMVERHp1szsYuACd7/Roq/cmsCZ7r5Kg5vWLShjSURERKTruRh4FLjb3V82s2VaWO5mouj52SX7cw9gp+lt3MwWApZz96/UzWqui7LqKImISE93EXCUmf2CiJMYcHRjm9R9KLAkIiIi0sW4+ytmdhdw4XQW3Q8418y+TxTv3m0Gd/EaMKh0M/yIqKN3JeqiLCIivZC73wDc0Oh2dFfqCiciIiLSy5jZnsBX3P0X5f28wDhgRXd/u6GNExERkW5FGUsiIiIivc/NwI5mdjtRpHtW4BwFlUREROSLUsaSiIiIiIiIiIi0ySyNboCIiIiIiIiIiHRPCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJJ8YWY22sy8mdfrHbS/YWX7C5b3o8xsbEfsS6S3MbOLyu9rjRlYdpiZec370e35W7TQt+b90NK2ddprHzXbntDCcczNbPRMbvv4sp3tZmDZJcuyi87MPqezj1+a2U0dtX2R7s7Mljeza8xsYnldbWbLNrpdtcxsrJmdXvP+M8dLEWksM1uxHDveMbM3zewGM1ul0e3qKGbWr+79KDP7d6PaI42nwJK01Y3ubnWvL3XEjtx9bNn+Kx2xfZHeqlwUfBN4G5huEKSeu49092Ht2KQngZ1rtj+h/Pbvbsd9NG17aNOxC/gPcFTNsWzkTG5+O9r4nYpI5zKz5YA7gYeBxYGvAOOBO81sqUa2rZa7D3P3vWsmfeZ4KSKNY2arAv8EXgSWAZYijim3mNmSjWxbRzCzDYCPa6e5+yh371IBeelcCiyJiPRew4A3gPOAEY1tCgDd/ul7uUldGPg1sLUyCkS6vN8Dt7n7Ie7+hru/7u6HAbeWeV2Vji0iXcepwL+BH7n7K+VYciDwAHBoY5vWIfo0ugHS9SiwJO3KzJYraaCvmdkHZvaQmW1ZM3+Ymd1hZlua2RMlXfSEMu+gst4rZrZ33Trewv7GmNnxddNOMLMjO+ozivQgI4Brymv5+qfz5bc33swmlS5vi9fN/0y3VDMbaGanmdnrZvaWmV1rZkvUrXOwmb1gZu+a2d/MbCkz27z8xocA55WuYUvWdIUbamabmNkUM1uobntPN3WVM7Olyz7fL/s41cwGtvXLMbMhZvYXM3uvfJ7RZjb3dFYbAdwIXAkMBjas2+bKZnZ3+U7vA1aumTfczD4xs/lrppmZPWulq6KZLWtm15fP+LyZnWJmc9Ysf4eZ7WBmZ5V9DK2Zt5eZvVyOu2eY2Wxt/W5EegIzWwT4BnBKM7P/CGxuZguUbmg/KMeUieV3tF/NdpqOVcvX/L7vMbOVazdoZnua2ZNm9pGZPWpmI2rmzW9ml5nZG+X4ONbM1qqZP7Ycc5s9XpZlFjSzS8pv/NXy9wLt+62JSK1ynbMOcLa719+vHEFkMk3v9z+0XLesZWbjynXH+WbWz8x2Kef7t8xsVN2+3czWNbO/m9mHZvaYmX29Zv5AMzu9bHuSmT1jZgc2s40VyjFmQpm2rpndbNGl710zu8vM1i7zLgNuqVn3jvL3p9eEZnahmd1Qt5+NzOzxmvc7m9njpd3jzey7X/zbl65EgSVpNxZP5v9GdCtZjripOgnIZjZ7zaJfBr4DrAdsDvzQzC4FZgOWAA4CTjWzpWdgt5cDO9a0wcq2L5npDyTSg5XfyjZEUOl24B1qspZKcONa4C/AgsAhwMHT2exfgQWA1YHFiCf+tzQFMMxsL+AA4PtEVs+twPXAzaVL2rPArqU72lN1274FeAXYoaaNawLm7neX9t4G/AtYFFiTOAaNnuEvpYZFQGos8BYRUFsbWBq4cDqrbgdc4+5PAk9Q0x2ufA/XAw8BiwC789nv9CbgdWD7mmnrAJPd/T6LOnO3AXeU9dcmvu9z6tqwP/AqsIi7TyjTVgbWB1YCNgKGA8cj0rutWv4d18y8xwFjWvD3AOAR4viyL3CCmX2zbp2TgH3KMo8B15rZAAAz+yHxm9sHmA84EbjcptWQOxWYlbh+Whi4GbjR6oLZ7j6mueNl2c8/iG64SxPdcf5X2mBf6FsRkS9ixfLvo/Uz3P1Odz9rBn7/AHMABxLXZmsCmxDXHOsRx6EdgUPM7Bt1uzkJ+BWwEHAVcHVNQPlM4t5qQ2AgcY90kJlt1Mw2ziYeMg4m7ueuBYaW7V4HXFY+007AxuVvc/f1m/lOLgY2Kdtqkoj7NsxsF+CE8n0MBvYijqlbN7Mt6S7cXS+9vtCLuFHzZl5zEBdpfeuWfwNYo/w9DJgMzF0z/8/AQ3XrvAqkmnW8Zt4oYGz5ex7gI2CdmmXvbfR3pJdeXf0FrEUETfqW95cBd9XMPxJ4rG6d3Vv5LQ4DpgAD69Z5AhhZ/n4GOKRu/o3AsuXvCU3LlvdDy7FlaHn/W+COmvknAL+uacu9ddteuHb9Vr6LfwOj6qb9EHgNGFAzbdmyvVVb2M4i5fg2b3lfBZ6r+/5eA2atmTa8bHPR8v7kpu+05jMfW/7+Ze1/ozLty3Xr3wHcXrfML8t/69r97krUR5it0f8v6qVXo15EjaIpRIC6ft6s5be1PRFkvrRu/mgiKF57rBpeM38g8C7w/fL+ReCgum1cBFxX/h4HnFA3f/6av8fWHqeaOV6OJAJJVjOtL/ABMKzR37VeevXUVzmOOLB0K8tM7/ffdAxZvmb+b8s1Q/+aafcAB9a8d2CvmvezAE8Dh5f3SwFfqtvvlUClbhvH1byflbrrnHI886ZtUXdvVqaNYto1Yd9yPNqjpl0vA8uV9xOAferWP4Sa6x+9ut9LGUvSVs0V734fmARcVFI2Pyrp2oOJoFOTl939nZr3E4mngLU+IDKYWuXubwNjiAg8wPdQtpLIjBgBjHH3T8r7a4G1zWzh8n5FojZArf+2sr01iQuHiVYzwhpxUbN0yQAaCtxbu5K7D3f3GR1F5GJgXZs2gtq3KU/Qyv7XqNv3i2XejGQ/1luVCKxNqmnrv4kbxZVbWGc74G53f6O8vxZYzKaNuLci8Ki7f1SzTv13ejGwgZktVLIMduCzn3Gdus/4bJlX+xnvaKZt/6nb771AP+q6N4r0MhOJ49bgZuYtWP5t+j3/p27+/cDyddM+Xcbd3yUC68uXjMqF+fwx9V6mHU9OAPazKBOQS3eVAV/gs6wJzA9MrTk+TCaupdpyDBSRGfNm+Xfe5mbO4O+/Se310ETgCXevLZLd3P1R7XFnKvAg045NrwGHmtm/LUqUOPAtPntfBjXXDeVaYVaL0iYvm9nk0haaWa9Z5doyM61XyYbAq+7+uJnNR3TlPaXueuZYdKzq1hRYknZjZvMS3VX6EWmcgzzStV8h0smbTG1m9cnNbXIGd3058O3SzWQE027CRKRl2wHblz73k4gUaAO2LfP78vlzRGu/yUnAs80EnM3dD6lZt81dMtz9fuJGLZVucB+7+4M1+z+/hf3/rQ27+5Dmz5Gz0PLN3nZE4KfpOx1Tpjd1MZzud+ru/yIyu75NdIN7z90fLrMnAee08Bn/UbOZ92fg8zUFFJutXyfSSzQdP5obEnxV4trkwWbmQfyWm7ueaW6ZD2ve1zLK8cTdLyFutkYRx4CdgMfNbPXp7KPJJODWFo4PZ87gNkTki3uAOJeuUD/DzDYlMqChld9/kxIYqtWW+6PaY9OfiDpyPya6xxtxn1S/jU+vG8xsGaJr/kvA14lspaZ2fpFruEuAjUsg6duUbnBETxOAjZs5Vi3c7JakW1BgSdrTBsTTsl3c/UF3/8CioOSC01lvZl0NzE087XvI3V/u4P2JdGvlomFBYA53H9D0IupzNNUEeozP32y1lKkDMTz3omb2pbp9jTSzJd19IvA8UX+pad4sZnZbaQ9El5TpuYTop/9pX/2a/X+mvWY2u5n9xNpWwPshYLmm+ihle8sRT+vuq1/YzOYhahgtVPedXsBnv9Plzax/zarNfae1n7E2UD6eaTVhmvY7R/mM03uK+OW6OiurExeST09nPZEey92fJ2q9/biZ2T8ErqzJsF6ubv5KRB2mWp8uU36TXwEeL9lLz1D3+yWyjO4zswFm9i3gLXe/xN0PIn6jzxHHgebUHy/HAyuYWb+aNpiZ7VPqs4lIB3D3V4hu/Xs3U8/sYOJ6q8Xffzs0ofa4Y8BXiaD0PESdpoPc/SZ3f6scH9ZqYTtNtgA+cPe93f3xkrldX0dputdr7n4XkVWdiCypy8v0iURXuPrrmXVUY6l7U2BJ2lNTsd1kZrOVp2ynEymiHTYspbu/RxSV2wd1gxOZESOIfv31T8IuJp4uzUMUkl3cYrTGuc1sOLBnSxt091uJIuCXWIxcNqeZHUDUCGh6Wn8i8DMz26AEe44lgtFNx46ngY3KTVZLWUGXEBdFcYliqgAAIABJREFUI/lsYOkUIrB1opnNV26kziG6x7433W/k8y4mAi+nlO0tSWR1Xe/uzV0IbgX8y91fb2Y7y1sMRnAh8RTx12Y2yMzWJQoCN/cZ1yXqNtR+xpOJ/yYnlDYtBJxHZDZ8MJ3PMxmolv2uStRdOqmue5xIb7Qv8A0zO9bMBpvZvGb2ayJQfVDNcpuZ2Y8tRlkaQfw+T6rb1rFmtmYpWPsHovvIFWXecUDFzL5RtjGS6Op6LPH7/BVwusVolP2J4riL0nLGVP3x8lLieus8M/uymQ0ijr8VpnVjEZGO8VNgSWLwofnKq0oEoH9F67//mXWAmW1qUej/aKLY9lnE7/5/wLZmNpeZDQHOJ4JCrd2XPQEMshiBcnYzG0YcC71mvQmAm9lmZtbfzFra3qVEzc4X3L226/8ooovetmUfXyOOlYt90Q8vXYcCS9Ju3P0R4AfAYcTFzWnAoUSWQkenNl5GFKK9soP3I9ITbEeMHFLvT8QFxzfd/QWiW9xI4sLkF8RoY63ZgSjOeBeRQr0xsKG7N9U6Opk4LlxRllsV2Mrdm558HUw8pX+HFi62PEaLuxd4xd0frZn+BvFkbmXiKf94oh7SFu7+hbt7lYDLcKIu1HPA3URx3ZayB1r6Tm8mPuuIkrXwTaLWwCvE97Ff/QqlltODxIVYbe2E18pnXKO0aRxxrN1yBj7j/xE3r/8lMtOuBo6azjoiPZ67jyd+k2sRv6uniRHV1nf352oWPZ24SXwZ+CNRQPe6us0dSQTlXya6xWzVFLx197OIzOpzibpNPwG2cffbyzHwG8DsRAbDO0Rg6iB3b6l7/2eOl6UOy3Aiq/JR4re+MHEMnl7gWURmQjlvb0BcMzxNZDMuRfz+nm/t998Ouz+SuEb7H5EZtI27/690q/sWsFqZ9w/g78TotC3el7n79cT1wTnESLVHEFmdE5vWK9meBxMPz96h+e7ElPnzUVemxN3PBw4nHji+VfY1yt1P/WIfXboSa8P1tkiXY2Y7ATu6+4jpLiwi7cLMjgHWdvfNGt0WEZGOYmZjidGKRrUwfyjR1WVxd5/QWe0Skd6tFL3e2N3HNrotIn0b3QCRmVFqGMxFDFF5YIObI9IrlG5sBmxEZM2IiIiIiEgvpa5w0t2dT/QFvs7dx0xvYRFpFycRXbmmELUDRERERESkl1JXOBERERERERERaRNlLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItEmXDCxZ+JWZ3WNmt5vZGDMbYmZfNrNvdcD+RprZ7tNZ5qftvV8REelZqmaLVs2uqZrdUzUbVzXbtxP2ObI6nXNYG7c7ofw7rGo2ur23LyIyo6pmQ6tmL9RNG1U1G9mgJvV4ZuZmdquZ3WZm483saDPrkveOItJ4XfXgsCOwKrCOu28A/Aa4GPgKsE1778zdR7v7OdNZbL/23q+IiPQ4VwJnV9zXAr4GjKiajejIHVbcR1emfw4TERH5ojZx9w2B1Yj7sMMb3B4R6aL6NroBLVgQmAIYgLvfbGYfAKcBC5rZDe6+hZktDJwCDAL6Awe5+x1mNgqYDGwILABc6O4nmtlQYDQwAOgHHO3uf7V42jEUOAo4Efh62f/dwIHAdWW/Y4G9gGeIYNeKwEDgbHc/w8yGAXsAfYDX3L3Dn1SLiEjXUI1zwMcV978CVNw/rJodCpxYNbsTOBuYB5gE7Flxf7ZqtiVwJPAxcG/F/WdVs6WBU4lzySDgwor7idVmzmEV97+WJ/ZDK+6jqmaLA38E5gA+AfaquD9VjfPiJ8D6xDn2gor7b6tmcwBnAIuV7T4DfLfiPrWZz7cJ8KOK+/bl/aHA+Ir7Ne32JYqIfEFVs3HAucAI4rp814r7w1WzrwHHA1OJY+kJFfdLqma3EceyR6pmZwG3VdwvrJptRjzYHg2cVbY1V5n/07KvCcT9yDYV9/WqZusCxxDH5InEMffFzvrsncXdP7HIwH3CzH7j7h+Y2eHAJsBswBh3PxLAzLYBfk6cw54H9nL3iWZ2N3AVsHlZZ093H29m/dF9lUi311Uzli4EFgWeN7OLzGwXYBywP3Hg2qIsdz5wrLtvDCTgHDPrU+atBWwJrAvsaWarlvVvdvd1gK2B5er2uzKwjbuv4u6rAxOAwe4+DHjF3Ye5+3+IYNOz7v51YD1gpJktX7axFfB7HfxERHqdFYEH66Y9DHwV+B1wQyWe/B5FBJvmJYI621YiO3dg1WwrYCXg4Eqc29YE9qiazUo5h1VaPocBXAT8ruK+EVAt75tsXNZbC9i3ajY7sCRwXcV9o4r72kSAa1gLn+8WYNWq2dzl/RbAmBn4XkREOtJA4MNyzDwcOKRMXwkYWaZvCBxWpl9LHL8gjodblb+3Am4Algf+WNZbgzjufaUsMz8wuQSV5gH+AGxfju1nEQ+oeyR3f5MIFC1jZjsD85V7pHWAFc1sczNbgrhP2szd1yce0jf99/gScf80DDi0b9++l+ac9xk+fPhD66677mK6rxLp3rpkxpK7v1ECQWsT2UMV4GfEgQoAi6esGwInmtmnk4GFyt9XufsU4AMzu75sZyxwmpktCNwH1HcdeBpwM7sauAO4yd2fb6aJ25Q2NHXLm4M4Cb0OTHD3u9v40UVEpPvq08K0ycBwYB+AivudVbNHiHPYAxX3V8qy+wMfEU/Ij6ma/b6suzBxMzMWOK3awjmsajYQ+GrF/W9lPzdUzS6pmg0qi1xZcZ9clp0AzAuMB7YoT/CdyN79e3MfruI+tWp2FbBDaf+4pu2JiDRQH6JkBsDjxPESIvB9VMkC/Zg4vkEElv5YnXa9v27VrC+wasV9fNVsAHBkNTJypgDLAosQ9wmzEdlREA+vFwOuqsa9yCx03Yf27aUvkQG2DbBs6c0Bcd5alugu92XgxnJ/1h94rmndiy66aFzO+drLL798kV133XW5d99995SnnnqK1157rV/NtnRfJdINdcnAkpmtCTxcDiR3m9mvgceIjKImfYBPStS7fn0o3ehqTHX3q8zsLqIrwDrAsWa2YtMCJU1zBeJEsSpwqZn9yt3Pr9tWH+CH7n5P3X6HAe990c8rIiI9wiPA9nXTViOymFbjs+elPsQNSO00K9POIG6OhlXcJ1fN7gf6VNyvqtadw6o157BWTCn/flA3vQ/wI2AjolvH21WzE2k+QNbkYqLLwmN8NhtKRKQjvUt0N6s1O/AOQMW99vjWdAz7E3AmsFsJjL9Rln2sarYQcbz+GxEo2Qt4oqx3DBF4/0bp0nwlnz0uNl3r9wHuq7hv2Q6fr8szs8HEA/wnic/+S3fPdcv8BLjW3X/U3DZeeumlZ4cMGTIVWK4M1sSUKVN83333fWvFFVd8CPgz8K+U0iTdV4l0L101qr4tEfRpat+8RO2HscBgiCAQMK5plDgzW9jMxtass4OZ9bF46rAZcKuZHQks6+5XAgcRJ6OFm3ZaAlpHuftYdz+J6GO9Upndr6ab3Rhg57JOPzO73qKuhYiI9F43A3NXzbYDqJrNRnR7O77M+1aZvilRQ+kuYPWq2Xxl/dOJ7NolgJtKUGkD4skt1XIOq7RwDqu4vws8VjX7Rll+M6IG0sRW2rwEcFcJKi0AbNfaB6y430/UaFq7oqfIItJJKu5vAB9Uo0dD0/F1Y+CBVlZbArixBJW+Q3SZa/I34KfAP4gMpmOIbnBN640tQaVlaLl78F3AiqX+HVWzLapmf/jin67rs8jo+h1whkcQbwzwnaZ7IzMbbWbrAzcBw61kyprZXmZ2QNN2DjzwwO2AbW+99da/DB48eMqcc87JCius8M7tt98+BNjpk08++dsxxxzz5m9/+9t799tvv8MGDRo0f855wc7+vCLyxXXJjCXgCGAUcKeZTSKi4nsRT337l+Jv+xPBndPNbD/iacNB7j61ZCw9A9xI9Oc9z90fMLN3y/L9gDmB7O4Pm9nqZb8PAN81s3uIIqcTgd3KvPOAB8zsNOBY4OSS/TSVKDL3jJkN6cDvREREurCKu5cR4E6vmh1GnLtOr7j/vWr2GHBu1ez7RPHu3Srur1ajGOp1VbPJwO0V9zFVs6nAKVWzN4H7idpGg4BLyrY/PYeVArWr1zRjF+CMUlj7PeD702n274FLS7Drf8AVZV+t+RPTsqBERDrL/wGnluPlLMBvyyAIzS48yxxzHOiTJ99c7d//f8DVwJNVs0EV97eIYNKqFfd3q9EFy4igCESQ6eyq2e7Af4mC0587LlbcX6+a7QpcVo37lbeBH7Tj5+0Kbi4P7RcivrMjy/RziADcP83sEyJL6Q4Ai/Pf38rAS88Be9dsb/Edd9xxLDBg7733PgLYa4cddnh41113feWOO+5Yec4553x7+PDhry+66KJT3njjjeXnm2++BYHxOecXifpODxBZTY+mlHQeEulCzN0b3YZ2V0aFm+DuoxvcFBERkU6Xcz4YWIbIgronpdRuJ/uq2Z+BSsX96fbapojIzMo5b0cEnwaX1zxEEH7zlNI9ra0rHc/MJrj70NppOedVgIVSSjfUL59znpUonv4tInN30fLqD7wAvEh0y7sauD2l1Fp2roh0MAWWREREepic8zZE9tHHRAbvY0TB2ZtSSlPbss2q2XpEfaWbKu5HtFdbRUTaQ855NeAaaroIA9eklLZpYRXpRM0Flr6InLMBiwPfJGoDLkYUVZ+fyLh9kQg43QJcD0xoz4cqItK6HhlYEhER6c1yzn2Bhyj1mYpngdVSSm82plUiIh0n5zw/EVRoOu69AGyYUnqmca2SjpRznosyAAXRNa8pq+kj4r//88CjRPe5+1NKHzeoqSI9ngJL7STnvAgxvOaCRPR8IWAB4LiU0n8b2TYREel9cs6XATuWt68A30wptVboVkSkW8o570zUaJ0LeAtYFhidUtq1oQ2TTpVz7gN8FRhBjMa6GBFoGgi8BuyZUhrTuBaK9FxdtXh3d1QhCozPVjPtGeDnjWmOiIj0cqcCWwEvA3MDZ+SchytjSUR6ipzzQKKb73Ci3s5ORGBpLLoG73VKQe+HyguAnPNCxP8fm9dOF5H2pYyldlL6/f6FSMVsGp7i38DIlNK/GtYwERHplcp56UGimO2KwIlEDYrNU0qvNbJtIiIzK+e8KXAS0VPgT8BPUkoflXkDUkqTGtk+EZHeRIGldlRGL/gHsC7wJjARmBd4BLgMOKPphCciItKZcs47ACcDrxLBpVca3CQRkS8s59yfOJZ9m+jm+7OU0o2NbZV0d2bmwG1EgsA8wFXAKHdv04AXIr2NAkvtLOc8L5F++w7RBaFCjF6wDFE49U7g6JTSc41qo4iI9E45562B04iuIluklF5ocJNERGZYznll4BxgaeDvwK4aZl7aQwks9XP3T8ysLzAaeNLdj2psy0S6BwWWOkDOeRlgaNPTk5zzLMDWwI+BlQEHxgN/IIZB1X8EERHpFDnnzYCzgXeJgt4TGtsiEZHWlWvpo4HdgA+AX6aURje0UdKj1AaWyvvBwBPAl939AzM7HNiEqKc7xt2PLMttQ9Tz6kOMQreXu080s7uJrKfNyzp7uvv4zv5cIp1FgaVOlnMeQoxasS4whKjDdB1QTSm908i2iYhI75BzHgacT9ygbZNSerKxLRIRaV7O+cvARcQoX3cD30sp/a+xrZKepj6wVKY9SAQzVwDWcvd9zcyAK4EziYLx5wObuPtHZrYfsJC7H2xmTwGHu/ulZrYpUHX3VTr7c4l0FgWWGiTnPAD4ATF6xYrA68B9wLEppXGNbJuIiPR8Oed1gYuBj4ERKaXHGtwkEZFPlQEIfkSUlegLnAKcoEx/6QgtBJYeBnYGDgWWJWroAswFXECcPw8Gni7T+wPPuftOZjYBWNzLzbaZvQks5e5vdMLHEel0Cix1ATnntYFfEE9i5iKKfZ8PjE4pTW5k20REpOfKOa8BXA5MBXZIKWkoZhFpuJzzYOLGfSPiuniXlNITjW2V9GStdYUDLgQud/dct85PgOXc/UfNbG8C8JWm4t8lsLSEu7/VoR9EpEEUWOpCSuHvg4HNgKWI6PdtRD/ylxrZNhER6ZlKMdw/AbMAO6WU7m1wk0SkF8s5bw8cCwwmbugPSil90vpaIjOnmeLd5wLPu/uhZrYnsCWwg7tPMbPRRK3CN4BrgTXc/S0z2wuYy91/UwJLh7j7JWa2IfAHd1+5EZ9NpDMosNQF5Zz7ADsAewErAZOBh4ATgZuUAiwiIu0p57wcUWS0P7BzSunOBjdJRHqZnPPsRN2arYAJwA9TSnc1tFHSa5TA0m3EQ5aFiIBRpQSaZgF+BWwMfAJc6+7HlfW+A/yMqFn4HLC3u79fAktnEQkDA4AfuLuygqXHUmCpi8s5LwUcDnwNWBh4DPgr8LuU0nuNbJuIiPQcOecliAvp2YGRKaVbGtwkEeklcs7rAacSA9v8Fdg7pfRhY1sl0nZmNsHdhza6HSKdRYGlbqI8xfkxsD0xMsErwD1ENzkVXBURkZmWcx5KBJfmBvZMKY1pbItEpCfLOfclMvK/Swxkc3BK6a+NbZXIzFNgSXobBZa6mTJCxobAAcCqwGzAw0Q/30vVB11ERGZGznkR4AZgXqIrytUNbpKI9EA552WJwWqWB24hCnSrsLGISDekwFI3lnNeADgE+DqwBPAU8A/gVymlVxvZNhER6b7K+eVGYAFg35TSFQ1ukoj0EOUh6aHA3sRw7SeklE5vbKtERGRmKLDUA5Q04u8BuxHFvt8HHgR+A9yuYt8iIvJF5Zy/RASXFgV+nlK6qMFNEpFuLue8EHAxsBZwHzFYwAuNbZWIiMwsBZZ6mJzzCsBhxAl7fuBRYhjpP6oIooiIfBE550FEcGkocEhK6ezGtkhEuquc825EptJswBnAMSmlqY1tlYiItAcFlnqonPNAYH9gW2A54EXgbuDolNJTjWybiIh0HznnuYExwFLAESmlUxvcJBHpRsoxZDSwKfA4sFtK6ZGGNkpERNqVAks9XOnHPhzYjyj23Rd4CDgduFJPikREZHpyznMSBb2XJ0YjPanBTRKRbiDnvCVQJbLoLwN+mlKa3NhWiYhIe1NgqRcpI/0cRowqtzjwBNHF4dcppTcb2TYREenacs6zA9cDKwPHp5R+3eAmiUgXlXOeFTgVGAG8AOyXUrqlsa0SEZGOosBSL5Rz7g+MBP4PWBF4G3gAOC6ldE8DmyYiIl1YznkAcC2wBvC7lNKoxrZIRLqanPMawJnEiMXXA3uklN5vbKtERKQjKbDUy+WcVwd+QdwkDAYeJlKVz0wpfdTItomISNdTMhH+CnyNGBjikAY3SUS6gJxzH+A44sHlRKIm22WNbZWIiHQGBZYEgJzzPEAF2BJYFngWuIMYseO5RrZNRES6lpxzP+AvRNfqs1JKP29wk0SkgXLOXwEuJLrK3g58P6X0emNbJSIinUWBJfmMnPMswDbAPsAqwFSi2Pcf4P/Zu+8wu6qqj+PflYTee4cgvQSpEkAgNGnS4wZfEIKodEEZQaVF6ZiA9I6hw6ZXIzUgXUIgIMVGFFCR3gIhwHr/WHvI5TIzmZnMzJny+zzPfTL33HPPWfdmzp5z1tl7bW5LKekXRkREyDkPAK4FNgEuBQ7U3wiRvqVMEvMTYpIYgFOB09QWiIj0LUosSbNyzksARwHrAYsTU8TeDoxMKb1bZWwiIlK9MvTlKmDL8u/euqAU6RtyzvMBlxPnieOA3VNKL1UblYiIVEGJJZmqUqx1b2Bnotj368ATwHEppaerjE1ERKpVerpeRvR2vQEYpuSSSO+Wc/4/YDgwO/A74PCU0ueVBiUiIpVRYknaJOe8DnAYsDpxMvEMcAlwSUppcpWxiYhINcpwmIuBocCtwG66yBTpfXLOswIXAVsAfwN+mFJ6stqoRESkakosSbvknOclEkzfApYB/gHcDxybUvpPlbGJiEjXK8mlc4BdgdHALimlz6qNSkQ6Ss55Y+A0YBHgeuAAzSAsIiKgxJJMo1JfYyiwD7AKMIko9n0KcLeGQ4iI9B0luXQaMAy4FxiaUvq00qBEpEPknG8HlgAaUkqjq45HRES6DyWWpMPknJchin0PBhYGngNuImYH+aDK2EREpOvknEcAPyKmHd9eQ6VFer6c8yxAv5TS+1XHIiIi3YsSS9Lhcs4zAwcCOwErAv8FHgeOSSk9X2VsIiLSNXLOxwEHAI8C22rIjIiIiEjvpMSSdJoyJGJD4GfAqsBMRLHvC4CrVHtDRKR3yzkfCRwCjAW+nVL6qOKQRERERKSDKbEkXSLnvABwOLARsBQxk8g9wPEppderjE1ERDpPzvlQYF9gw5TSv6qOR0REREQ6lhJL0qVyzgOA3YDvA4OAD4FxwEnAQyr2LSLS++Scp08pfVJ1HCIiIiLS8ZRYksrknFcGjgDWAuYH/gxk4BwNlxARERERERHp/pRYksrlnGcDfgJsC6wAvEIUe/1VSukfVcYmIiIiUgUzOw7YB1jA3T+tOh4REZHmKLEk3UYp9r05cDBR7Ls/MB44G7gxpfR5heGJiIiIdBkzG1t+/Lm731VpMCIiIi1QYkm6pZzzosQwufWBJYG/AKOBk1NKb1UZm4iIiEhnMrOVgGOAx4jzoAzs5+5DzWwR4H53X7qsOxrYBhhWHp8BMwA/dPfxZjYc+BTYDLgYuAr4DVHrcjbgQnc/r8s+nIiI9DpKLEm3lnOenij0vRtxAvQO8CQxm9yfqoxNREREpDOY2VHAv4jE0hhgceIm21LA94DfAoOB14FzgAQcDRzv7p+Y2cbAMHffvSSWvgNs4O5vmtkRwER3P8XMpgfuB/Zy9+e68jOKiEjvocSS9Bg55zWAXwBrAnMDzwBXAhemlCZVGZuIiIhIRzGzJ4FvufsbZvZX4AfA/sBZwI+ApwEHXgNmdPfzzWxD4BDiHGk64AN336QkluZ094PLth8vu5lY/p0b+LW7X9c1n05ERHobJZakx8k5zwkcCmwJLAf8E/gjcExK6eUqYxMRkfYxs0WJnhcLANMDF7v76Z28z2FAf3e/qIO3O8HdB3bkNqXvMLOliUlMniiLlgNuA/4ErEzUodwVuIbosdQAfAw8C2zt7k+Y2SDgDHcfUhJLA9z9iLL9scC+7v44IiIiHUCJJemxcs79iJnk9idOsj4HngJOB+5IKemXW0SkhzCzx4hhPDeb2UzAHcDp7n5jxaG1mRJLMi3M7DDgNXcfVZ5/A7gBWA14gaiJdJiZPUAkjNY1s1WA64kklBPnQoOaSSwdB8zm7j82s+mAm4H93f2lLv2gIiLSa/SrOgCR9kopfZ5SuimltBmwFnA7sARR4PKJnPOvc86zVxqkiIhMlZkNAT5x95sB3P0j4HDgUDOb38xuMbMHzOxOM1uivGcrM3vMzP5oZqeUZcua2d1mdp+ZPWVmh5TlA81sjJk9amZjzWy7snxYuejGzJY0szvM7H4zu6f0GsHMhpvZEWY2umzzp2X5LGZ2eVn/MTO72sx0XiUdYRvgpsYnpWfRO8AywIvAneWl0cDDZZ3xRDL2CeBuonfTXM1s/zhgZjN7hKjfdK2SSiIiMi3UY0l6lZzzjMA+wM5Ed/HXiZOsY1NK46uMTUREmmZmBwLLuPuPa5bNBvwbuBX4o7ufY2brAT8B9iZ6qK7l7v81swuIXhczAhPKUKDpgPFEj9aTgDfd/RgzWxjY3d1PLEPhBrr7cDN7CPiVu99pZlsCR7v74JJ4Wh/YAjCigPKKxEX+iu5+VYn3UmCUu9+rHksiIiLSlyixJL1WznldohbTGsR0us8Ao4BLU0qTKwxNRERqmNnBwNfqEktzAv8ghvUs7e5vl+VzABsAP3D3xp5HswCTgNmJKdpXBSYTQ4dWJv4OnEMMJ3oCuM3dX29MLAEjgVfcfY6a/b8NfA04CPifu59dlo8hZuV6BTgM2KrEOBA4wt0vU2JJRERE+hJ12ZZeK6X0cEppe+LC4kLiguMMYFzO+ayc80KVBigiIo2eJdrqWqsD44hzFatZ3r+JZVaWnQe8CQxx9yHA34ji3DcRyaZ7id5GT5vZPK2I67Py78S65f2B/YANgW3dfUPgurJcREREpE9RYkl6vZTSGymlBuIiZS9ieNwuwNic8x05501yztbiRkREpDPdA8xhZtsDlOLdvwJOLK/tWJZvCpwNPAKsYWbzlfefC2wMLAXc7e6TzWx9IomEmR0NLO/u1xO9jN4FFm7cubu/DzxnZpuV9b8FjHf391qIeSngEXd/x8wWALaf9q9BREREpOfRUDjpk3LOywJHAoOBhYDniEKZp6WUPqwyNhGRvsjMliISRHMRPX/OdffzzGwR4GKiftLHwPfd/VUz24lIEk0majD9vCSERgBvAWOBFYgi4B+UbU8HzArc6u5H19VYWpbo8WRl/f3c/V+lxtKEmhm6xgDDiOFvVwGfAq8Bfydm8jpVQ+FERESkL1FiSfq0nPMswAHATsSd7f8AjwO/Tim9WGVsIiIiIiIiIt2dEksiQBkKNwRoIOp8zEAU+z4fuCal9Fnz7xYRERERERHpm5RYEqmTc14Q+CWwEVFD42/A3cAJKaXXq4xNREREREREpDtRYkmkGTnnAcSU0nsCg4iaG+OAk1JKD1UZm4iIiIiIiEh3oMSSSCvknFcGjgDWAuYnpsbOwLkppY+qjE1ERERERESkKkosibRBznl24KfAt4nZhl4hpr3+VUrppSpjExEREREREelqvT6xNMLsOGAfYIEG90/buY2fNLif2rGRSU9Win1vARwErEpMjT0eOAu4KaX0eYXhiYiIiIiIiHSvvbLwAAAgAElEQVSJvpBYGlt+/HmD+13t3MaEBveBHReV9CY550WBI4H1gYHAi8Bo4OSU0tsVhiYiIiIiIiLSqXp1YmmE2UrAMcBjwJIN7vuMMBsDDGtwn1DWmdDgPnCE2TBgJWBZYFFgTIP7ISPMbgY2Bx4Fjm5wv7/rP4n0BDnn6YG9gN2AlYG3gSeB41NKT1QZm4iIiIiIiEhn6O2JpaOAfxGJpTHAQsC9NJ9Y+hkwmJj96xFgjwb3F9VjSdoq57wm8AtgDWAuotj3FcCFKaVPqoxNREREREREpKP0qzqATrY9cFuD+/PAe8RQpZbc3uD+fkNk214gZv8SabOU0hMppZ2A1YCzgZmBEcBTOefzy/A5ERERERERkR5tQNUBdJYRZksDiwOXjzCD+KxDgc8BK+vMWPe2iXXP+3dymNLLlRpLv8g5Hw5sB+wP7ABsm3N+GjgN+H1Kqfd2HRQREREREZFeq9cmloCdgIYG91EAI8y+AdwA3AOsCLwE7AtMbsW2Jo4wm7vB/a1OilV6uTJL3I3AjTnnJYGjgHWBa4EXcs63AqeklN6rMEwRERERERGRNunNQ+G2AW5qfNLg/jjwDnAecPQIs/uIHklvtmJbpwMPjjAb3glxSh+TUnoppbQn8HXgCCK5eQgxTO6anPOgSgMUERERERERaaVeXbxbpKfIOa8HHEbUZJoVeAYYBVyWUmpNrzoRERERERGRLqfEUifJOQ8APlPtHGmLnPO8xGxymwLLAP8gZjQ8NqX03wpDExEREREREfkKJZY6QM75XGBpYKbymBmYHVgtpfRalbFJz5Rz7g8k4EfAKsDHwFPASOA+JSxFRERERESkO1BiqQPknDclhi0tUrP4npTSptVEJL1Jznk5otj3N4CFgOeA41JKN1camIiIiIiIiPR5Six1kJzzD4ETgHnKoteIHiano+nkpQPknGcBfkz0ZDohpZQrDklERERERET6OCWWOlDO+Xjiwv/vwDhgHWAx4Hng98CIlNI71UUoIiIiIiIiItJx+lUdQC9zOHAjMDqlNAxYFfglMIlIOD2Vc74x57x2dSGKiIiIiIiIiHQM9VjqYDlnA/qllD6rW74W8HNgdWBe4FngWuCclNJHXR6oiIiIiIiIiMg0UmKpi+Wc5wR+CmwNLA+8CjwKHJNS+muVsYmIiIiIiIiItIUSSxUpPZs2Bw4Cvg7MAIwHLgauSil9WmF4IiIiIiIiIiJTpcRSN5BzXoioxTQEWBr4G3A/MaX8fyoMTURERERERESkWUosdSM55wHAd4G9gEFE0e+ngdOAP6SU9J8lIiIiIiIiIt2GEkvdVM55WeAIYDCwCPACcDswMqX0bpWxiYiIiIiIiIiAEkvdXs55JmBf4DvASsCbwJPAiSmlP1UZm4iIiIiIiIj0bUos9SA557WBw4DVgbmBZ4GrgfNTSh9XGZv0DmZmwHHApsRQzA+BvQEH1nT3G9q53YHAKHcfMg2xLV4bg5mNAYa5+4T2bnNamJkDDxDfDcBb7r5jB2x3ODDB3UdN67ZEREREpGuMMFsUOAdYAJgeuLjB/fQRZqOAUQ3uY+rWHwIMa3Af1rWRinS8flUHIK2XUnospbQjsBpwOjAjcCLwdM55VM556UoDlN5gZ+L3a7C7rw/8BrgC+BqwbZWBdZMY6m3i7kPKY5qTSiIiIiLSY10PXNjg/g1gHWCHEWY7VByTSJdQj6UeLOdswFbAgcCqwADgGeAC4JqU0mcVhic9kJkdTPRW2s7dPyvL1iHuviwIjHP3Lc1sYeBMYC7ijsxh7v5g6W0zGdiAuFtzmbuPLD2WLidqhS0PfAR8391ftriL8ySwK1FXbFzZ3/zE7/TB5d/6GMYANwHblTgO9nInyMyOBDYBZgJGu/vRjb2miIL4i7j70Gn8rhyYzt0/rVs+ocS6rbuvZ2brAscA0wHvAXu7+6tT+Q4nAEsB/3P3M8p2bwd2d/c3pyVuEREREelYpffRMQ1xY7Zx2brASOBFSo+lEWbbA0cD/yau2xZscB82wmwO4AxgUWAW4IQG95tGmA0DViHOnx9ocD+xCz+WSKupx1IPllLylNLtKaUtgDWADMwHXASMyzmfnnNeoNIgpae5jPiD9rKZXW5mewBPEcmd0e6+ZVnvEuA4d98ISMBFZta/vPYNIuG5LvBDM1utLF8TON3dvwncQPzxbLQDsJG730XMgniXu29I1Be7GhjbRAwAC5UYdgFOAjCz3YD5yrC7wcAgM9uirL8G8OC0JpVq3GNmY8pjUFk2PzC5JJXmLJ9zJ3ffgEj6jizrtfQdQvQU27V8pkWASUoqiYiIiHRLg4ibo7WeAVZufDLCbB7ipuIWDe5bEzcdG40Ebmpw3xjYDDiurA9xnnuAkkrSnQ2oOgDpGCmlV4H9cs7TERejw4DdgJ1yzk8DpwJ3p5TURU2a5e5vlkTQ2sDGQAPwU+DQxnXMbBaiR9LIKMkUi4GFys83ld5OE83sjrKd64Fn3X18Wecq4NiaXd/s7hPLz1sAB5R4xpvZe0Th+qZcVv59kUjoQAyXW770aAKYnbjL8wLwmbtf24qvorU2qe+xRPSSurj8vC6wGHBT+a76Af1a8R3i7i+Y2XRmthQwFLiyA+MWERERkY7Tv5llk2uerwuMbXB/rTy/A/he+XlbYNkRZj8uzw1Ypvz8eIP7Pzo4XpEOpcRSL5NSmkwM9xmVc16BGFr0DWLI0PM559uAU1JK71UXpXRXZrYW8Iy7Pwo8amYnAc8BX69ZrT/waVOFuEuSxOoWf17+rU1qflb3/IP6TdU9b25Y5wcA7u42JUPTHzjW3XNdbAOb2E9nadxPf+AJd9+qLpbZafk7bHQlkSj+FjG0T0RERES6n2eBneqWrc6XezHVjxaqPenrDwxtcP9f7QojzJan685fRdpNQ+F6sZTS8ymlXYmkwNHAp8AhRLHv63LOa1QaoHRH2wHHmVlj2zAPUSR+DDETIe7+HvCUme0IYGYLl6Fgje8Zamb9zWxGIiFyf1m+rJktVn7+P+D3zcRwJ+UPs5mtVPb/PDCxMYapGA18t3FYmZmNMrNvtuJ9neERYijewBLLlmZ2Riu+w0ZXETXUnnf3SV0Yt4iIiIi03j3AHKWGEiPMZgJ+RUy01OhRYI0RZo3ns7WlGUYTo00YYTbbCLMHRpjN1vlhi3QMJZb6gJTSxJTSiJTSYGBzonjxWsC9OeeHcs4H5JxnqDZK6SaOAj4EHjKz+4DrgL2Juy3Tm9mjZjaY+MO3t5ndT9QCOszdG3smvQT8gfjj+Tt3f7IsPwU41cz+CHybGGLXlJ8A25dtnw4kd5/cRAzNuYgYGvewmT0EvOjuD7b9q5h27v4GsCdwdRmatzdRyBta/g4b3/9vIql2eZcFLSIiIiJt0hAzYu0A7D/C7AngYeDyhqgf2rjOa8RN/rtHxHnhKzWbOAhYb0ScJ98BnNTg/v7U9lsmcxKpnGaF66NyznMDPyPq2SwH/ItoAI9JKb1UZWzSczXOaObuoyoOpcfIOS8InAdcCIwuw1kBMLOZgbuAb7oaaxEREZE+K+e8PTF784LAHOWxELB+SmlChaGJKLHU1+Wc+xG9Rw4gprLsD4wHzgeuSyk1V9tG5CuUWGq7cgw+DSxL9PZ6Ebhq9913n2/SpEl7Ake7+61VxigiIiIi1co5HwwcyZdLQzyeUlq7opBEvqDEknwh57woUex7feBrwF+Ae4ETUkr/a+m9ItJ+OefLKOPqiw+AI1NKv60oJBERERHpZnLOPwcOA+Ysi14jJtq5EfhdSkmFvqUSSizJV+ScpyOmvhwGDCJq7jwNjATuSynpl0akA+Wc1yaGvDUWacwppZ0rDElEREREuqGc86+JmkwTiUlvVgWWYUqS6XfAzbXlFUQ6mxJL0qKc80rA4USx74WJxuoW4LcppakWlBORqSuFF59kyknBdMCeKaV7Kg1MRERERLqdnPNIYHBKab1SVmEIsC/RKWAgUV5hPHAm8KA6BkhnU2JJWiXnPAtRh2lHYCXi4ncscHxK6akqYxPpDXLO55QfTyBmA5kD2CuldGd1UYmIiIhId5Rznjul9FbdsumBoUSJhRWB+YjyJo8Dp6aUXujyQKVPUGJJ2qT0rFgPOJTodjk78CwxXfrFKaVJFYYn0mPlnPullD4vPy8K/B6YC9g7pXR7pcGJiIiISI+Sc54d+AGwDZFkmh54AbgPOD2l9N8Kw5NeRoklabec8zxE8bhvETNa/RN4CDhWU16KTJuc84LAH4B5gf1SSjdXHJKIiIiI9EA554WJukwbAssDHwHPAzcTnQNU4kSmiRJLMs3KuN7tgP2AVYB+xJjec4EbUkqfVRieSI+Vc56fSC4tCByQUrq+4pBEREREpAfLOa8I/BRYk+gc0Fj0+1Li2k1Fv6XNlFiSDpVzXhw4AvgmUTjur8DdwIkppdcrDE2kR8o5z0sklxYGfpJSurrikEREupSZDQQedPdFa5YNBya4+6hm3vPF62Z2NXAIMTHCKHcf0oZ9T3D3ge0MXUSk2yolToYQnQPqi36fDTygot/SWkosSacoheP2AHYnGqoPgKeAEcD9aqREWq8MO/0DsBhwSErp8opDEhHpMtOaWKrbjhJLIiJ1yrXbjsS1W2PR778ypej38xWGJz2AEkvS6XLOg4BfAmsRQ3qeI8bznpZS+qDK2ER6ipzzXERyaSBwWErpd9VGJCLSNVpKLAEHAxcDOwCzAXu6+zN1PZbGAMPKW0e5+xAzmx+4Efieu//DzH5IzKI0PXEj7EB3/9TMJgArA88AS7v7Z2a2HrCjux/SmZ9bRKQKpej3XsC2wArAjEQ9pjFE0e//VBeddFdKLEmXyTnPCvwY2J7IhP8XeAI4PqU0vsrYRHqCnPMcRHJpKeCXKaULKg5JRKTTTSWxdCRwortfYGZbAru7+3dbSiwBWwO3AT9z9yfMbH2i3shO7v65mZ0KvOju5zb2WCrD6Ua5+2gzOw84193HdfZnFxGpUs55IeL6bQiRZPqYSDLdAlyUUnqvuuikO1FiSbpcGc+7AVHvYDXiDuMzwGXAqJTSJxWGJ9KtlbtIo4lii0emlM6pOCQRkU41lcTScGBFd59Y1rvI3TdpIbF0OVGodjFgcEkknUzc9Pp3WWdm4AF3b6hJLG0LfIeYuvtud1+/sz6viEh3lHNeAfgJMQplWeB1vlz0W9dwfZgSS1KpnPN8wGHApkQDNQF4CDgmpfSvCkMT6bZK77/RxJ2j4SmlMyoOSUSk05jZPMBz7r5AzbKTgUeAUxtrINXWUGohsfQ3YCgwGHje3S8xs5HAa+5+chP7bkwsTVfeexSwkLuf2BmfVUSku6vpJNA4I/iSxDXcM0TR7zGqp9v3KLEk3ULOuR9RH2Ff4OvA58SMBOcAN6WUPq8wPJFuJ+c8C/B7ovbHsSmlUyoOSUSk05jZS0Rdo3FmNhPwAJEgur+5xNJ000338ieffHJRXWLpUnffwMzmJuqFrEVcIB0HbODuH5vZscCz7n51bfFuMzu37HM1d3+5Kz63iEh3lnOejriG24ModbIA8Bei3MkpKaXnKgxPupASS9Lt5JyXIGomrAcsQTROdwEnpZTeqDI2ke4k5zwzcAeRjD0xpXRSxSGJiHSKUgfpZGAy0A84y92vMrMJ11xzzZLAkk8//fQOl156acPIkSOfveyyy74+22yz/evKK69cs6ni3WWbRwEfuvtIM/sJ8H9E/ZCxwCGlUHdtYmlj4Eh336jLPriISA+Rc54N+D6wHdGrfibgBeB+YtKmf7fwdunhlFiSbqtMe/l9YpaWlYH3gXHACOCP6mIpAjnnGYHbgdWB36SUjq84JBGRLlWKy44FFqpZ/BqwQUrpLx21HzM7GHjH3Ud11DZFRHqjnPOCwEFE0e/lgUlEkukW4EIV/e59lFiSHiHn/HXgl8CaRBfLPxPTBJ+RUvqwythEqpZznoGY4Wgt4LcppeHVRiQi0rVyzj8DfkXcIQe4NaW0bUds28xmJ845JhIzx6lArYhIK+Wclydm3qwt+v08MXHTdSr63TsosSQ9SulieTCwDbASMYPLE0SNmT9XGZtIlUoPv1uAdYDTU0pHVhySiEiXyDlvApwFzFMerwMb67xARKT7KEW/vwnsT5RxaCz6/SxTin6rrm4PpcSS9EilYRpCZL9XA2YhZiK4FLgkpTS5uuhEqlEKKN5M1Cc7O6X0i4pDEhHpNOVc4BjgR0QyaT/gN8AbKaWtqoxNRESaV85ZtwP2ZErR778Sw5pPTSk9U2F40g5KLEmPl3OeH/g5sAmwDPAS8CBwTErplSpjE+lq5Q/1DcCGwPkppYaKQxIR6XA55zmBq4lE+kPALimld8oEIANSSn+vNEAREWmVnPOsRIJpeyLJNDNRj+kBosTDqxWGJ62kxJL0Gjnn/sQ0wD8CVgE+A8YTXStvUddK6StyzgOA64CNgYtTSgdXHJKISIfJOa8NXETc4b4AOFwTeoiI9Hw55wWAA4kOA8sDnxD1mG4nbpi+W2F40gIllqRXyjkvCRxJ1JtZAngRuBM4KaX0VpWxiXSFkmjNwLeAS4ADdeElIj1ZGfp2GDHT0HvAASmlu6qNSkREOkPOeTmitu5gouj3G0SS6Qogp5QmVRie1FFiSXq1MlvWD4DvAoOAd4EniRoMD+tCW3qznHM/4CpgK+KP8L76nReRnijnPAtwObApMWlHSim9Xm1UIiLS2cpNhXWBA4ii318D/kkU/T4HuFcjU6qnxJL0GTnn1YFfAGsA8wF/Bq4HzkopTawyNpHOUpJLlxMzKV4D/FDJJRHpSXLOqxCTcyxOTE/905TSZ9VGJSIiXa3UEt0WGEbMEL4g8DemFP0eX110fZsSS9Ln5JxnJ7pVbkMUiPs38DhwbErp+SpjE+kM5U7PpURRxOuBPZVcEpGeIOe8PzFBxyTg0JTSDRWHJCIi3UAp+r0HsCOwAjFL+IvAH4kkkyZx6kJKLEmfVS62NwF+AqwKzAQ8A4wCLk8pTa4uOpGOVX7fLwK+A9wE7KFuwyLSXeWcZwQuBr5NDHdIukgQEZGmlFnCDyCu7VYAPiXqMd0BnJdSeqfC8PoEJZZE+GIGgl8Qs2gtA/ydyHYfqykupbcoyaXzgV2A24BdlVwSke4m57w0UR9uGWKGy311s0dERFqj/A05BFgbWA54k0gyXY6KfncaJZZEapSZtBLwQ2AVYDIwHjgTuE3Dh6SnK8mls4HdgN8Duyi5JCLdRc75e8Ax5elRKaVLq4xHRER6pnLOOxj4MV8u+v1n4Fzgbp0DdxwllkSakXNeCjgCWIcoGPoCMBr4TUrp7SpjE5kW5Q/t6UThwzuJISYqhCsilSkFWc8BhgJ/Bb6bUvpbtVGJiEhvkHMeQNTX/T5RY3chouj3k0Q9pqcrDK9XUGJJZCpKnYcfEsOHBgFvE43QSSmlR6uMTWRa5JxPBfYC7gN2Sil9WnFIItIH5ZwXBTKwMjFM9/sppY+rjUpERHqjnPMswO7ATkQ9ptmIDgQPAaeklF6uMLweS4klkTbIOa9JzE6zBjAvUVD0OuCclNLEKmMTaY+c82+AvYEHgB1Ux0REulLOeUfgZGAG4MSU0lkVhyQiIn1Eznk+YH9gMyLJ9BlfLvqtUSqtpMSSSDvknOcAfgpsTTRC96eUtqo2KpH2yTmfAOxH3KnZPqX0ScUhiUgvV2oangJ8j6h5sUdKaXy1UYmISF9VyqAcQtRlWpYYpfIcMZnE1epJ2zIllkSmQalV8y3grZTSn6qOR6S9cs6/Bg4maogdM7X1RUTaq9whvpbo/XsPMUPlh9VGJSIi8sX13TeIot+rAksB/yKKfp8H3Kmi31+lxJKIiACQc96a+GOp4XAi0ilyzpsBZwBzAL8FTtaMqyIi0h2Vot9bEzVJVwIWJop+n5VSOrfK2LobJZZEREREpNPlnDchinT/B/iBJsAQEZGeIuc8MzF8exfgzJTS9RWH1K0osSQiIiIina7Msvpb4LCU0rtVxyMiIiIdQ4klERERERERERFpl35VByAiIiIiIiIiIj2TEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEk0g2Z2UAze6Vu2XAzG9aObU3oqLhEzMzN7H4zG1MeN1Qdk4i0Tjl+B9QtW9XMzig/t+nvjJkNM7PhHRuliIhI72Zmi5rZrWb2uJk9ZWY/LstHmdmQJtYfYmajujrOthgw9VVERES+ZBN3/7TqIERk2rn7U8CBVcchIiLSh1wPHO/uN5vZTMAdZvZy1UFNC/VYEulhGrPaZnafmT1hZoPK8lnM7HIze6D0JFm1iff+sPQ2ecTMzqm/cy3SXmY2wcwOM7OHyvN1zeye8vt4m5ktUpYvbGY3lN/fh8zsm9VGLtK3lR6yY5pYPtTMrrIwh5ldamb3mtljZrZ93bp7mdnImufnN/5tEhERkSlKj6RP3P1mAHf/CDgcOLRuve3NbJyZ3Q5s0eWBtpESSyI9z2zAR+6+EXAk8Muy/HDgVXffANgDOLf2TWa2PrAVsJG7rwN8DPygy6KW3uSemqFwjReP8wOT3X09M5sTOAPYqfw+XgA0XnReAhxXfn8TcJGZ9e/qDyAizTOzDYEfAcPc3Ynj9yZ33xjYDDjOzOapecv1wA5m1s/MZgSWc/dnujxwERGR7m8QMK5u2TPAyo1Pyt/YM4Et3H1rYLquC6991FtBpOfpD1xRfn6euKAH2JySKHL3f5rZ5nXv2wZYCbjXzABmBj7r9GilN2pqKNxMwMXl53WBxYCbyu9aP6Cfmc0CbACMLMsBDFgI+FJNMRGpzMrEDYuL3H1SWbYtsGxjDQjiuF2m8Q3u/o6ZPQVsBMwL3NqF8YqIiPQkTd1Q7Q9Mrnm+LjDW3V8rz+8AvtfZgU0LJZZEuqf3+WpmembgXQB3n1izvLFx6kec7DeqP777Axe6+8kdGKdIrQ/Kv/2BJ9x9q9oXzWx24FN3H9LVgYlIq20DDAFuN7Pz3P0d4pge6u7/q13RzJaveXoFsCswD7BvF8UqIiLS0zwL7FS3bHW+3IupfmSZ0c1pKJxIN+TubwITzWw1gFLUbSPgyRbedg+wY1l/WeCWutdHA0PLMAXM7Fgz26WjYxcBHgEGmdlAADPb0szOcPf3gKfMrPH3dOEynE5/i0S6j5Pc/d9EF/xflGWjgd0AzGy2Ujtttrr33UYMk5uuvF9ERES+6h5gjsZ6heU671fAiTXrPAqsYWZzl+dDuzbEtlOPJZHua3fgbDObTCSBTylD3Jpb/xiiXs0DwOfA3rUvuvtdZrYy8Ecz+xgYC1zbadFLn+Xub5jZnsDV5XftHaJeC8TF6blmdhDxe3qYu39eUagifdE9Zubl50eA8+pebzweLwHGmtkZwEHAeWa2Q3ntRHd/v/bvkbtPMrP7iBNmERERaYK7e/l7eq6ZHUH0Cj63XKvtWtZ5zcwOAe42s/eAu4Glq4t66ixqMoqIiIiItE/peXgf8G13f7/qeERERKTraPiBiIiIiLRbGd76GHCFkkoiIiJ9j3osiYiIiIiIiIhIu6jHkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLPUxZjafmZ1tZv80s4lm9ryZNZiZdYPYRpnZqDa+Z7omtjG6QwMT6UPKMeQ1j4/N7GEzG1x1bK1R3yY08fqwus/3qZm9aGZ7dsb+yz52aeV7J5jZz9uwr+FmNqaNIYp0uZp2ZasmXhtjZkPauL3646xNx05HqD9nqT/Wzaz/tJ5bmdmxZbtDW7HuwLLuwPJ8uJlNmJb9N7GPdrdvIn1NV51PtXTeU7f/+sfwjoxDRImlPsTM+gG3AcsBWwLzAj8G9gVGVhhau5jZ94C/1C5z92HuvkVFIYn0Fn9wd3N3AxYAHgZ+b2bzVBxXi5pqE5oxqebzzQwMB843s02mcf93A4fXLiv7ubo173f3ge5+4rTEINKN/Rs41cymn5aNmNmFwAW1y7rDsVN7rJvZ4sAnwBLTuNkdgHeA7dsRz3B3HziN+//CtLZvIn1Up55PNXVc1mrcd9n/JGDPmmXDOyIGkUZKLPUtywHfAA509+fcfaK73wXsBKw2rSd7FegPVN7TSqQ3c/d3gZ8DE4Fh1UYzVW1uE9z9E3e/CrgROHga9z9gGt8v0puNAOambxxn/ZjGc2wzW4ZITJ0AbD213phdoCd87yLdViedT+m4lG5DiaW+pX/590tdMN39KXffyN0/aWo4WhPdvSeY2bZmls3sQzN7ycx2rnvPN8zsfjP7yMz+UbpzT1+3jQ3N7AYz+8TMvtIwmtk2ZvaImb1rZu+Y2V1mtmx57VHgd8ASpTvn5dnv93AAACAASURBVPWxmtkfzeycum3uUTtUzsx+WuL/0MweN7PNWv91ivQN7v4p8Dfga/DF0JWdzezccowvXZbvZjG89iMzG29m/9e4DTOb38yuNrM3zez9so1v1O6npePRzIaY2X/NbHUzG2sxlPdeM1uivN5km9AGz9d8voXN7Mqyv48shsrtURPLQDP7wMyWNbNxZvY/M3NgQ+Dosv9Ny7puNcN8zOzbZvZs2e7TZrZ5zWsTzGxYzT7czFY0s0dLF/rHzezrzX0AM5vOzI43s1fLd3y/ma3Rxu9BpLO8DRwBHGFmCzW1goWfmdkL5Rj5r5ldZGazmdmM5TjbA9ijHB8/KO/74tgpz1c2sztLO/E/MzvNzGaoeX2Mme1vZieZ2dtm9h8zO6gulj3N7KlyrL9hZteb2YLNfbjGY93MTgReKotfMrNXzGxwOYbnqFl/ztIeztzMJncA7gKuBeYENqrb3yCLc6SPzWwssHrd68OsZihca9oHM/u+mf29fG8Pm9laZrb81No3M1uq/Lx23fbuszJUzswWLO3qu+X/5EozW6C571OkN2rifGpdM7vHzN4qx+UjtcdROQYfqGmHrrY4n2r2uGyrsq07LM69XjazM81s1prXHzSzvc1shMX12L/N7ICa12c1swvM7LWyjces5tymrLNHTbv+tGkIba+kxFIf4u7PAg8BF5Y/9r80s/XNrP/U3tuE44FrgIWIu5BXmtkqAGa2AnA3cDkwP7AJsB5x163WMcADwHylof2Cma1InEydXvbxNaIhPq98lsHAnsA/S3fO3ZqI8Qpgx7rPl4DGrupHA3uVZfMQw2GytXDhJtIXWSR+lwb+UbP4QGKIxmLu/jeLxMvJwP5Er4S9gZPNbJuy/tnADMAKwMLAPcAfGi+0Wnk8zgycCvyAaBdeBS6EVrcJLVm+5vNdD3xOXKjNQfSwOM/Mlqz9WoAziQvlRUs38/uBX5X9312/AzNbiegZdRbRNh4GXGNmy7cQ16nEd7oo8Bxwm5nN2My65wPrA5uW7V8I3G3NXMSLVOAC4O/ASc28/iOibdmDOPYGAysCP3X3j8txdglwSTnOLqzfgJktQhyLY4njZnNgM+JYqvUT4H1gSaLn9nFmtl7ZxubAb4FDibIBKxM35+rPY77C3X9etgmwpLsv6u6PAv8lkkWNvgOMdveJzWxqe+BWd3+JOPa/GA5XkmR3AH8un3FYibUlLbYP5TOfRbRL8xPnW3cAb0ytfXP3vwOPEe13Y4wLAGsAt5Y2617ib8ayRA/614j2TD3Ppc+oPZ8ys7mBO4kyJQOJ85rbmXKd0r+89gfimPw6cR51tbu/0JrzjlbEsyBxLfYgsAiwNjFk76K6VX8GvFHi3Jk4v2tMgB1LnNutCcxX3nuLTan3thdwHLAPcX64H3CKmW3Z1nilm3N3PfrQA5idaACeA7w8XgVSeX0UMKruPV9aBkwATqhbZwxwUc3619a9vi4xtnf6mm1c1dx+gFmBQXWvDwI+qHk+DJjQwjbmIWocbFqez0mcRM5JXKB+CGxd9/7z6z+/Hnr0pUc5hkbXPJ+LuMB6E5i3LBsDPFz3vgnA/nXLfgmMKT8/BZxc9/r85d+pHo/AkNJeDal5/evAezXPv9ImNPH5hgEf1zyfEdgN+BTYuCxbGZit7n1jgaHl54Ellr3r1hkDDK9b9kXMRI+qO+tePxnYueY7HFa3j81r1p2ttGHfK8+H13y/SxLJsJXqtn9nfUx66NHVj9KuDCs/f7P8rq5Tno+pOUYWAL5W994DgdvqtjWqbp3aY+c44hzHal7ftBzjC9bsc0zdNm4kElgQ7d5yda9vAzzbXBx1x3rj8Tuw5vWT+HLbOoa6Nq/mtQWByUxpI08kztWsPN8DeAuYqeY9m9Tuk5r2sDXtA3AfcH7d65cCG9bEO7zu9drP/GPg5ZoY9wOuqYnltbr/kwHEkKAhVf9+6qFHZzyYyvkUkSRare49s5Xjal7iesWBrWpen4ma85OmjssW4vm4sZ2sWXYs8EjdssXLfhctzx8E7q5b51bgx+Xn24ib+bXH9/w1P7/MV8+Xjqrfph49/6EeS32Mu7/n7ke4+4pEr4HdgdeBq8xsUBs29WLd87HEXUWAtYChVjPzANFTanriZKvRgy3E+QGwmMXwt9fN7FNgPDBLawN09zeJLH/jML3tgXvd/R1gJeJi9ra6OH9I3E0T6cs2rzkmXiESOEPc/Y2adR5q/MHM5iNqgZxZdzwdx5Tj6WTgIDP7i8Uw2kOJpA607Xh8rObnd4kkdFvNULOP94k79Du7+73l9beJu2l/txhm4kTvpfr2p9k2rBmrAH+qXeDuh7r7NS2858Wadd8nipOv2MR6axK9qJ6t+w43Q22adCPu/iBxR/70Jnqr/A/Yzsz+VIZMfU70XG71335gNWCce1y9FH8iehytXLPsMb7si/bE3d8G1rYYUv+WmX0G3NLGOOpdDWxiMTvvYsRxfGcz624HPOHu/yvPbyPO2Rp7CAwCnnP3j2re8/cW9t2a9qGp9ml3d7+/dR/vi17s65bnX/QQJ84L5wc+r9n3ZOIiWe2T9GbNnk+5+yTifOQWi+G4k4H3yvtmKdcrZxK9f/5kUepjVyJJ3FHWAgbXtQv/LK/VHpvNtpdEsmxbYIKZ3WhmRxJJNEqPyEWBc+v28St07Pc6Siz1IRb1CeZrfO7u/3H3y4ANiN4CzdUXak3ByH5Maeg+Zkq3zPpH7YxNH7YQ64bAzcAjxInUzMRQlba6khgON4Dodt54Afdx+XfJJmJct8ktifQdf6g5HmbxqMH2TN06tcfvpPLvRk0cTwsDuPuVRPJpOFF/ZBfgeYsaH60+HusupKB9Bfwn1Wx/Oncf5O7Xwxfd1O8mTni+R9x1M+DRJvbVbBvWDGtnvLVq29paH5fl/Zv4Dv+vifVFqvQzYujE9+uWHw4cTVxMLUv0atmHth03H/HV89vG989Yt169fhD14ohhe9cRCZcZgC3aGMeXuPs4YrjtUOC7wPXuPrmZ1bcH1iyJ7Y+JNgmmDKUbQPOfsSmtaR+mqX1y99eIIc47l+E1Xwd+X7P/+5s5Lzy/vfsU6QGaPZ8ys+WIY/vfwMZEb6XGNsoA3P1AIpF8IdHDbzjwlJnN1kHxfUyMOGnq2Ly3Zr1m20uPIXiLA4cQvUU3B54xs62Ycn63fhPbX7yDPoN0E0os9S1HAbebWf3/+wdEF/H3iDv1cza+UMb3DuarVqh7vgpR/BaiZ9FqtS+a2UJmtk8b6jntADzl7ke5+z/c/ROi+3ytz1qxnZuJnlLfIeo83VKW/5VoJOvj/JaZbdDKGEWE6AlJDEWpP54GWxThn9HMdgTedvcr3f0wovbGv4i72h11PLamTZialYkk9n7u/rC7v2dmcxK9qqZ1/88Sn/sLFkUyWypiuULNurMQ9eaeb2K9Z4gT0S/ViDOzncxstSbWF6mMu79K1Cs6nqil1GgocIG7X+Lur7n757T9b//TwKp1vaHWIhIr41oZ4lDgFnc/zd1f8agDWR9HS5qL8Roiqb4rcePrK8xsdmLo76LuPmPjA7iYKXWWngOWr6u31lJ9yNa0D021T9eXG30tfaZaVxDf3XeI+lCNF5XjgZWsZmY7C/tbCwXRRXq5LYGJ7r6Puz9fjpcv2hkzW8DMtiqvnefu+xFt2dI1603reU9T12yzmNmB5ZyjRWbWz8y+DfRz9+vc/XB3/ybwR2DX0vvzlSb2sY6ZbT2NsUs3o8RS3/Ibotvi9RazicxgZosTBRo/IIpl3w9sVg74OYAziDpF9YaZWbKYqWUf4iTotPLaCcC3LGZ2mdPMvkacbGzs7q1tAP8CLGMxu9wsZrY9UbC3MdkFcedvofJZZmyiSz0eRTFvLrHdVYbYURrvk4ARFjOazFQy69cQdahEpG2GA4eb2XZmNrOZrUO0KYsRQx6OJ7pCL2ExQ+RGRPfocR14PE61TWiFV4gk19DS9ixPtF9vM2VmzZb2v67FDClNFdg+lRgK84Oyzg7EcOSxLWzzOItZYeYm2uP3iO/1S9x9AlHU+EIzW6P8H+xO1HVScVzpjkYQ5x6r1iz7C7ChmS1uZnOb2c+IC6naY+8fwBpmNo+ZzdTEds8mhmQda2ZzWUws8lvgwpLQao2/ED2Gljez2c1sT2KoR2tvjv2XaEc2NbPpaxIqVxMFtOek+aG0WwFjSw+gWlcAy1pMbnIFcW52cvmMg4khvU1qZfswEvheacNnNbP9S6zjy+tTa98g6lTNCRzJlB7iAFcRNaF+V/5v5wJOARqYMvRHpK/5CzCXmW1RjskhxHHsRFszK3CdmR1U2sOZicT0JCK5DK07LltyOrCkmZ1sMUx3IaJd2IXoIdWikvxvAEaZ2TKlvVuLGOrbmMgfDhxZbjTObDFJwnVEsXDpRZRY6kNK1ngDYhjKTcTsHA8QJxWD3f1dd7+JOAG7iRiv/ypfPjlodAJRu+h/xEwk/9fYtbMMd9uSmGXltbKPJ4hhJa11PlE08rayjT2I4o8QdQZw94eAc8v2GwvhNeUKYpaCq+uWH0ucaF1GnPAcA+zh7re1IU4RAdz9EuJiYiSRhLmIKCh5dkkob0YMaX2CGJt/BnCYuzcel9N8PLahTWhpG28QJ1Q7l21cR1yojqO0PS04njgRfIOYFbN+2+OIITANZZ2jiILgf21hm0eX/f+H6DX1bY+6DE3Zj7hYvYOonfcDoujnk1OJW6TLld/jn9YtPpj4m/8sUV9sAaJdqT32zibOTV4mjvX67b5BDMXYmEjwjCZuMO3fhvCOIerIPUz0rFy/vH8B+2qv768oQ9z2Ji6o3gO+XZY/Vz7b1e5fqgFVa3viHKzeGCLxvYO7fwhsDaxDfMazgIOmElaL7YO730ycz51OnNt9F9iynDvCVNq3so33iaK+A6ipH1V6nW9O1Kj6M3F+uTCwgTc/K55Ir+budxC1hi5iyjnBAUSbsbDHbIubE6M4JhDnATsSx21jHaSpHpdTieF1ovD/mkRb9xRxDrZVC21UvR2Ycr33HtEb80wieYy7XwT8mri+fIu4xjtcw2B7H2v974xIMLMJxAXjqIpDERHplSym6X2JqDs1odJgRKTDmNkLxGQBT3fyfvYikvcqkCsiIp1uQNUBiIiIiIj0VmUo3ACi9/U7nZlUMrNZiXpSmwIvdNZ+REREamkonIiIiIhI51mdGF52ALBnJ+/rEGJYzDK0UHdJRESkI2konIiIiIiIiIiItIt6LImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosifQyZraEmd1qZk+Y2Z/M7Bft3M4oMxvSweGJiIiISB9iZm5m95vZA2Y23sx+bWYdeh1qZgua2bUduU0RaT0llkR6kfJH+mbgRndfE1gP2NT+v717j9d0Lhc//rmMcVZIDuUwOzpIW7s2HZwaZlTsKGl/EVtU0km1a6lUfql250eifm1KfiqHfHfJIYeNyRiKiGKLjlLaJSKMRszh+v1x34vVMsPMsp651+Hzfr3Wa57n+9yH6168nvU8131d32/Evt1GJkmSpElsRmbuCDwfeBpwxGgePDNvy8x/XZZ9IuLAiPjyaMYhTVaRmV3HIGmURMTLgC9k5jOGjD0feAYQwKHtv1cCA5m5MCKeBxwFTAHuAd6YmbdHxEnASZk5e/lehSRJkiaKiEhgamYuaJ+vA/wC2ATYCPgSzefQtYFvZOZRETEN+CLwe+A5wH1Aycx7I2Jz4Hge/uz6usy8OyJuycxpEXEgsCXN59+NgNmZ+Z723O8FCpDA3MzceTn8CqQJz4olaWLZArhh6EBmXgtcR3Nn6KXAtsCGwMERMRWowKGZ+RLgfODY5RqxpHGnF5G9iEt7EXN6Edf3Ij7aG+W2hl7EBj3bGiS1IuLjEXFnRKzYPp8dEdPan9mjeJ7ZETFltI6nR8rMu4BbgWcCWwHvz8ydgG2AN0bEyu2mOwNfzMztaT7LDlbgfx3oZeZ0ms+xH1rMaXYD9ge2BraLiGdGxLOAt9B8Fn4hsHJE/EMfLlGadEwsSRPLAmDRYsZnAhdl5n3ZlCmeBuwKPAuYl5k/bbc7FXj5colU0ng3Y6CPbQ0DmbcNLGNbQy/iwJ5tDdJE9XLgFmCnfp4kM6dn5sJ+nkMArEjzmfV7wEER8X3gIuApwHrtNjdk5uAN05uA9SJiTWDLzDy/HT8d+PBijn9uZs5tP/f+rD3mPTSVSqsCKwGr0FTyS3qcVuw6AEmj6nqadreHRMQLBx8O23bhEsYXl5iSpMUayFzQi3gH8ItexGdZTFvDQOZRvSW0NQxk3ttbTFvDQObdvYhbBjKn9RbT1jDQtjX0hrU1DNjWIE04EbEl8Fvgh8BeNAmIoVaMiBNobpjdD7w+M2+NiD2Aw2g+26wJHJaZsx6jVWqwnWp94Cvtfk8A5mTmv7fb/AQ4F9ie5n1rn8z8fd9+ARNM2wq3IfBL4Gs0SaPpmTk/Iq6h+Z0CzBu26xQeWRgRQ7YfanH73g7cTDMlxB+Bz2XmzSO9DkkPs2JJmkAycw5wb0S8AaAtJf4EsAawS0Ss1m66N3AOzR2c1SNii3Z8n3ZckpbawGLaGgaGtDX0hrU1DCyhrWFgGdoaehHP7C2mraFnW4M0Ee0FnN3+7LmYFcW2Bo5tW6bOAL7Qjj8b2L1t938Nf1/Z8ohWqWHHfDbwf9sWra2B50XE09rXtqRJNL0EOAV42yhc46TQtjJ+Hjg+M+cBmwEXt0mlHWh+70uUmfcAv4yIme3Q64APLOXpXwDcmZlbZObOmXnKyK5C0nBWLEkTz17A8RHxFpqqpFMz85iIuAOYHREPAJfSTMydEfFa4KsR8SBwG82XNElaVkPbGj7WizgGmM+wtoaBv29r2LjXtjUM/H1bw5mLOf65A5lzAXoRg20Nv+LhtoYHsK1BmqheBbw0M/8cEfcCOwx7/YbMvL59fBrwH+3j84AvR8RTaaYLeOqQfc7N9j0lHn5P+fmQ168APhwRR9B8nnpWu//NwF8y87/b7W6iSTzp0c1qE4IbAt/l4STf+4EvRsRdwDXAJTTVrn95lGMdABzX/re5q32+NH4O7Ni23T0AzAVOyExvqkqPk4klaYLJzN/RzJ80fPwE4ITFjP+Q5m7/8PED+xGfpImnt4S2hoHM+b3HaGtY581vfspdxx03dOxxtTUM2NYgTSjtCmCbACdHBDTfX14zbLOhy1wvBLKdgPsCmra4C9q5ef5nyHaLe08Z6mPAk4BdMvP+iPg2j96ipSXIzCUm/DPzQppK18WZPmS7k4Y8vhHYcTHHmjZ82/b5gdBMAA8cmZnHR/M/0yY07ZUbLM11SFoyW+EkSdKI9Ya0NQwMaWtok0qP2dawxk47fX2Ftda6p/c42hoGMrcYyNx5wLYGaSLaCxjIzJdn5stp2vn35O+rE58RERu3j19Ls8rtGu3PnHb8kGU872Y0cy/d37bJTR9h/Bo7vgPsHxGX0vx/cTrwkW5DkiYGK5YkSdJIzOo9SltDbynaGlZ44hPXADZa7/DDF/3pQx/6j94I2xp6w9oaBmxrkCaS3YFXDD7JzKsi4m7gxUO2+RxwdDvh9t00VUr3RMR/AFe20wF8HViZpfcx4IR23spf07Torv34LkVdyswf8cg2SkmjIJoVGCVJkpavWuvHeHii7kuAGaWUpf5g0mvaGn43kHl8b0hbw0CmbQ2SJEnLia1wkiSpKzsNefximmXBl8V3gP17tjVIkiR1xoolSZK03NVaNwV+BKxLM9nuH4D7geeUUuZ3GZskSZKWnnMsSZKkLqwJnEQzue5BwD6llB90GpEkaVKptQbwQuDmUsrtXccjjVe2wkmSpOWulHJDKeUw4Ms0S4U/veOQJEmTz1rAOTQLT0gaIRNLkiSpSzfTrOL0/K4DkTQ+1VpXr7Wu0nUcGn9KKX8BbgM26zoWaTyzFU6ahGqtU4BVSil/7ToWSZNbKeWeWuvdwFO7jkXS2Ne2Lm0C7EqzAMAmwNOAlwLXdRiaxq/bgfW6DkIaz6xYkiank4Frug5Cklp30bQjSNJjeT/wY+A/gQK8CPh5KcWkkkbqT8B6tVaLLqQRMrEkTU4LgKntXT9J6trdwNpdByFpXPgs8NMhz/8GfLGjWDQxXE1TsbR514FI45WJJWlyuh1YDXhC14FIEs3d4rVNdktaCvsC04C/tM9vBL7VWTSaCGbTTBGzXcdxSOOW5X7S5HQrTWJpXeCejmORpLcBUUrJrgORNDa1iececBDwa+AI4ESgllIWdRmbxr2baG66bgd8teNYpHHJiiVpcvotDyeWJKlTpZT7Synzuo5D0thUa12NZkn4NwGXATuWUi4A/hU4psvYNP6VUv5Gk1h6StexSOOVFUvS5PS/7b+uwiRJksasWusmwBk0K7/9J/C+werGUsr3u4xNE8rtwIZdByGNV1YsSZPTn4G/0sxRIEmSNObUWncBvkczsfLbSynvtWVWffJ7mpXh1ug6EGk8MrEkTU53AvOAjbsORNLEEhEZEZdGxJyIuD4iPhoRft6QtExqrYcBJ9Os+vaKUsqpHYekie1S4MnAc7oORBqPbIWTJqd7gftxjiVJ/TEjMxdExIrASTST7H6k25AkjQe11hVpJuV+FXAtsGcp5S+Pvpf0uF1J89l45/axpGXgHURpEmrLyOfRTOAtSX2RmQuAdwCHRsRqEfGMiLg4Ii6JiJ9ExHsAImJaRHw3Io6LiMsj4oKIeEL72uYRMSsiZkfEWRGxVju+R1sZdXlEnDa4vaTxq9a6DjCLJqn0bWCmSSUtJ7fQzLP03I7jkMYlE0vS5GViSVLfZeZdwK3AM4GtgPdn5k7ANsAbI2LldtOdgS9m5vbAdcC+7fjXgV5mTgcq8KGI2Ax4L/DSdvsrgQ8sp0uS1Ae11n8C5tC8VxxZSjmolLKg47A0SZRSFtEkltbvOhZpPDKxJE1eJpYkLS8rAotoJuE9KCK+D1xEs7Tzeu02N2TmDe3jm4D1ImJNYMvMPL8dPx34MPAyYBPgvyNiNrA3LkYgjVu11n8DzgZWBvYrpXyu45A0Od1OM4F3dB2INN44x5I0ec3j4S90ktQXEbEOzRLOvwS+RpM0mp6Z8yPiGmBKu+m8YbtO4ZE3wKIdnwJ8NzPf2rfAJfVd+wX+s8AbgF8Bry6l3NptVJrEfg7sAGwA/LHjWKRxxYolafKyYklSX7WTd38eOD4z5wGbARe3SaUdgGc/2v6ZeQ/wy4iY2Q69jqbl7WLgZRGxdnueQyLisH5dh6TRV2tdjaZK6RDgMmBHk0rq2CxgLeCfuw5EGm8mfWIpIjaNiHMi4kcRcXVEHP4Y29+yhPEjI+LAfsQo9clfgdVqrZP+fUDSqJsVEZcBPwPuomlfA3g/8MW2fe1VwCXA2o9xrAOA/xMRlwL/Anw8M28CPgRc2I5vD3xp1K9CUl/UWjehmU9pO+A/gVeWUu7vNiqJHwN30sz5J2kZTOpWuIhYATgLODYzT4yIlYDzI+KWzDyt4/CkfvsTTcXSWjRf/CTpccvMJc5NkZkX0kzgvTjTh2x30pDHNwI7LuZYpwH+rZbGmVrrTOA4YCXg7aWUUzsOSQKglHJnrfUOmupaSctgslcq7AKslpknAmTmg8BhQEbEvhHxg4i4IiKOjogpw3eOiI+1VU7fxklDNf7cSpNYWrfrQCRJ0sRXaz0MOAX4G/AKk0oag27HOUilZTbZE0tbADcMHcjMa2mWOT4CeCmwLc2kowcP3S4iXk5zd/XFNEsiP7P/4Uqj6hZgVUwsSZKkPqq1rlhr/RrN5+ubaOZTur7jsKTF+RPNynCTurNHWlaTPbG0gGb54+FmAhdl5n2ZmTSl9rsO22YX4IzMXNBWOl3c31ClUXcbsBDYqOtAJEnSxFRrXYdmUuQ9gTOAmaUUW/A1Vl1FU7G0edeBSOPJZM/EXg8cOnQgIl44+HDYtguHPV/cEsjSePJnmpXhNu06EEmSNPHUWp9L0/q2LvCRUspRHYckPZZLab4j71ZrnQb8ppTy825Dksa+SV2xlJlzgHsj4g0AEbEy8AlgDWCXiBhcin1v4Jxhu18CvCIiVmi32205hS2NlsHEkhVLkiRpVNVa96P5/LwysJ9JJY11tdYvA58HHgQ+AJyJN2ClpTLZK5YA9gKOj4i30FQlnZqZx0TEHcDsiHiAJnN90tCdMvPsiNgeuBK4G7h8+YYtPW5/Be4HntR1IJIkaWKotQbwGeCNwK+AV5dSbu02Kmmp/I1mft3B78i/oCkmkPQYoplCSNJkVGu9HvhVKeXVXcciSZLGt1rrasA3gZ1ovpDvXUq5v9uopKVTa50CzAa2b4fOL6XYlSIthUndCieJecDqXQchSZLGt1rrxsAcYAfgOOCVJpU0npRSFtJMgfKrduj8DsORxhUTS9LkNg9YtesgJEnS+FVrnQl8D1gfOLSUclgpxbYIjTullD8Ah9PMRXpqx+FI44aJJWlymwes9phbSZIkLUat9T00K789ALyilHJyxyFpkouIjIhLI2JORFwfER+NiKX+3ltK+RbNhPN3jkIs0yJi9uM9jjTWOXm3NLmZWJIkScus1roi8FVgT+Bamkm67+o2KukhMzJzQUSsSLMI0xHAR5Z251LKhf0KTJqIrFiSJrf7gNXayQolSZIeU611HeBimqTSGcBMk0oaizJzAfAO4NCIWC0inhERF0fEJRHxk4h4DzxUWfTdiDguIi6PiAsi4gnta5tHxKyImB0RZ0XEWu34Hm1l1OURcdqQ7beNiKsj4jzgwG6uXFq+TCxJk9ttNHMsrd11IJIkaeyrtW5FM0n3FsBHKZXLZAAAIABJREFUSykHllIWdByWtESZeRdwK/BMYCvg/Zm5E7AN8MaIWLnddGfgi5m5PXAdsG87/nWgl5nTgQp8KCI2A94LvLTd/krgAxExFTgZODAzdwMedzudNB7YCidNbrfStMKtSzNJoSRJ0mLVWvcDPgk8SDMHzcUdhyQtrRWBRTSTzH8sIo4B5gNPAdZrt7khM29oH98EbBwRawJbZubgCnGnA2cCrwM2Af47IgBWAn4HPAuYm5k/bbc/D9irnxcmjQUmlqTJ7RYeTixJkiQ9Qq01gM8AbwB+DexVSvldt1FJSyci1gE2BH4JfI0maTQ9M+dHxDXA4JQQ84btOoVHdvhEOz4F+G5mvnXYuZ67mO2lCc9WOGlyu43mbs2mXQciSZLGnlrrqjQVGm8GLgdeYlJJ40U7effngeMzcx6wGXBxm1TaAXj2o+2fmfcAv4yIme3Q64AP0Mwx9rKIWLs9zyERcRjwM2DtiNi83f41o35R0hhkxZI0uf2Z5u7MJl0HIkmSxpZa68Y0k3NvDhwHvLeUkt1GJS2VWRGxAk2l0neBD7fj7we+GBF3AdcAl9DMNfqXRznWAcBxEXEEcBdwQGbOjYgPARdGxDyaNrg3Z+YDEXEA8M2IuA84tx8XJ401kenfBmmyqrWuBvwCOKuU8rau45EkSWNDrXUGTTJpZeCDpZRvdBySJGmMshVOmsRKKfNoKpZcFU6SJAFQa30PcArNJN27m1SSJD0aW+FGWTu54frA7aWURV3HIy2F+2km8JYkSZNYrXVF4Cs0q1j9GNizlHJXt1FJksY6K5ZG33Y0Kw7s0HUg0lKah4klSZImtVrr2sBFNEmlM4AZJpUkSUvDiqXRtwBYFVuLNH78FRNLkiRNWrXWrWha39YDPlZK+WzHIUmSxhETS6NvHs3y7U/sOhBpKd2PiVBJkialWutrgU/SfH7dr5RyccchSZLGGRNLo+9+mj/MT+g6EGkp2QonSdIk084L+mngjcDNwKtLKb/rNipJ0njkHEuj736adjgTSxov5gKr1Vqndh2IJEnqv1rrqsCZwFuA7wM7mlSSJI2UiaXRZ8WSxps/0FQsrdN1IJIkqb9qrRsDlwI7AscDe5RS5nUblSRpPLMVbvQNzrG0RteBSEvpdzSJpXWBP3UciyRJ6pNa607Al4FVgHeUUr7RcUiSpAnAiqXR9zeaVrjVuw5EWkq30Kxk+OSO45AkSX1Sa303cBrwILC7SSVJ0mixYmmUlVKy1jofcL4ajRe3Aw8Am3QdiCRJGl211hWBrwB7AT8G9iyl3NVtVJKkicTEUn8swMSSxo8/07Rwbtp1IJIkafTUWtcGzgD+GfgO8IZSyoJuo5IkTTQmlvrDiiWNJ3fSJJae2nUgkiRpdNRatwJOAdYDPlZK+WzHIUmSJijnWOqP+Zi00zhRSnmAJrH0xK5jkSRJj1+tdR/gHJo5FPc3qSRJ6ieTH/1hK5zGhVrrc4Bnt0//sdZ6CvCDUsr/7TAsSZLGtV5EAnOApFl59X+BAwYy5z7KPkcCtwxknjTS89ZaA/jUHZ/73LvW3n//n6243np7lFJ+O9LjSZK0NKxY6g9b4TRefBw4HXgGsCVQaKqXJEnS4zNjIHP6QOYLgPuAN4zkIL2Il/Uiznms7Wqtq9LMo/TWJ7/73ReuuN56LzapJElaHqxY6g9b4TRevB64Anh6+/wXwMndhSNJ0sTSi1gZWB/4dfv8loHMae3jacBJA5nTh+1zAPAOYCGwJvCPvYjVgeOBjYFVgN8Arx3IXNSbMuXWtQ888MH7Lrlk03Xf9a4v//Gd73w2sF5v773vWOI+ET8BTgT2bM9x0EDm//TvNyFJmqisWOqPBZhY0jhQSrmT5kPlg+3Q90sp8zsMSZKkiWJWL2I2TTLnDwOZj1l1BNCLWAM4Fpg5kPlC4Hpge2Bz4NyBzJe04w8C02utO01Za60NFtx225M3/NSnDtrvHe9465DDLXaf9rU1gfsHMncCjgA+8HgvWJI0OZn86A9b4TSefIbmbuWmwH90HIskSRPFjIHMBb2IKcDpvYh3DWR+fin2ewD4K7BGL2IuTQIImgTTrr2Iwbmbpq25664rADPJTCIOLqWcPuxYj9gHuKh9bQrNqnEAN9GsHidJ0jKzYqk/TCxp3CilLALeA/yilPK7ruORJGkiGchcCHwd2LUdWtSLiPbxKovZfj5N8mcOMAuYNZB5KfBW4CXAHpucfvqM1V70onunbrLJXsDPF959921zzz33msWc/qF9BjJfAnyLJqE0eK6h8ypOQZKkETCx1B+2wmlcKaVcDuzUdRySJE1Q2wA/ax//jodXZH3j8A17ERsCWwxkPq2d/Ptz7UubAVdscvrpseDPf579wM03b7nwL3+5lqZiadESzrsZcMVA5t29iPWBV43aFUmaVCIiI+LSiLgkIn4YEYe040dGxIEjON77I+JfRj1QdcLkR39YsaRxp5SysOsYJEmaQGb1IpLm8/adPLwq3IeAk3sRd9O0om09bL87gLV7ET+gaYubB3wbOCZWWeWs2z7wgfdNWXvtKVOf8pRL76n17INPP31+b++9lxTDMcBpvYiZwJ+A/wLWHsVrlDS5zMjMBdHMBXddRFw20gNl5qeWZftoFjD4NvB/MvOqkZ5X/RGZ2XUME06t9WTgRaWUzbuORZIkSeNHL+Jg4GkDmYe3z58E/GTDo4768NSNNvowzQ3Mt5ZSLuwyTkmTSzSJ8qmZuaB9XoEzgGcBt2TmSRFxBPAyIGhWtXwt8EzgrZm5V7vfB2nmf/vnIfs9YpXKzPyfiHga8FWaVt31gfdl5pnL7aK11GyF648Hgam1VnvVtdxExKYRcU5E/Cgiro6Iw0d4nJMiYvoI9/33kewnSZIeMgvYphdxWTvp9nmrbbvtLVM32qgH/BmYaVJJUpeiadl9EXD1kLE1aZJJO2TmdsBJwJuAS4DnRcQT2013BS4Ydsg1gfvzkatUfgI4NTN3BN4JvLIvF6THzVa4/phL0wq3KnBfx7FoEoiIFYCzgGMz88SIWAk4PyJuyczTlmMo7wSOXo7nkyRpQtnk9NN/U0qZCVBrXRU4DZgBXAqUUsq8R9tfkvpoVlu5BPD2zPz14FoEmTk3Im6i+Q6yOvAE4EeZuSgizgReExE3AD/JzPkPr2EALHmVyjuAwYTUWn27Kj1uJpb64z6a362JJS0vuwCrZeaJAJn5YEQcBjwjIvYFDqUpSb0SGMjMhRHxPOAomjfye4A3ZubtQw8aEdsCH6NJlN4LHAL8od1vZ5q7ElcC7wXOBTaIiNnAIZn58/5esiRJ41+tdQWaO/9vBF5M83f31FrrRjRtJk8HvgK8p5TiHBaSujRjsBVuuIjYiuYG88zM/FVE7A7s1b58CvBZ4Ebg5MXtn4tfpfIM4DsR8Srgt4DdEWOUiaX+GKxYWq3rQDRpbAHcMHQgM6+NiPtpJrl7AfBXmrueB0fEV4EKvCozfxoRbwGOBfYZ3D8i1gK+QPMH5O6IeCVNQulTwB6ZuXm73WHAOpk5va2Qmt7na5UkadyrtW5B0+bxdGAasDrwa+CsWut04Ms0NynfWUr5ekdhStLSmgbc3CaVpgIH0dyYJjOviYgNgDUz8+3LcMz3Ay/OzJtGPVqNKhNL/XEPD7fCScvDAmBxSw3PBC7KzPsAIuI04PXA94F5mfnTdrtTgU8O23dbYGPgzLZUdYX252YgI+Js4HLg4sy8dXQvR5KkCe924B+ALYeM3UhTufR+4C5g71LKjzuITZKW1fnAfhFxFXA3TbXRrkNe/xZNt8Oy+CFwVkT8kWYe458CH83Mu0YhXo0iE0v9cS8Pt8JJy8P1NO1uD4mIFw4+HLbtwiWMD09MTaHpi95t+MkiYkuaxNPzgNMi4hOZ+bWRBC5J0mRUSrmz1no2zYpKKwN/A1YBPgpcB+xZSrmzwxAl6SGZOfy7w+D4kUOe7j3s5eOGPH4OMLC4/TJz2pDHtwDT23ma9gGe007zMZWme+IA4PMjuQb1j6vC9YcVS1quMnMOcG9EvAEgIlamKa9fA9glIgbbMvcGzgF+BqweEVu04/u040NdAfxjRExrj7lrRHwhIrYBPpKZszPzaJoVH7Zq95kaEa6GKEnSY6i1HkFzU+hq4Jc0N35eBJwNzDCpJGkiiIjtIuIHwA2ZefPS7peZf6WZw/Wydg7Xy4BNaabz0BhjxVJ/3EuTtFuj60A0qewFHN/Ol7SQZmnOYyLiDmB2RDxAs6LMSZmZEfFa4KsR8SBwG/CWoQfLzD9HxEHANyPibzQlrW8C7gRe25a5LqD5//317W7/D7g2Iv4zM4feoZAkSa1a62doFsS4EjicpiUd4OOllE93FpgkjbLM/D5Np8NI9nWy7nEiMl1cYrTVWrcGfgDsX0oxoypJkiRqrQF8Efg3YDbwTZo5DhcAbymlXNhddJIkjYwVS/1xPzAfeELXgUiSJKl7tdYVaCp7Xw2cB/yGJsn0G2CvUsot3UUnSdLIOcdSf9xPc+fJxJIkSdIkV2udQlOd9GrgO8BKwNtoKtx3NKkkSRrPrFjqDyuWJEmSRK11Ks2y2y8B/otmZaSnAycA7y6lOC+FJGlcM7HUH/NoEktrdh2IJEmSulFrXYVm1dUX0Kz2tj3NqsHvKqV8rcvYJEkaLSaW+mOwFc5V4SRJkiahWusaNEtlbwXMAWYAdwH7lFKu7TI2SZJGk4ml/pjf/qzadSCSJElavmqtawEXAJsDNwE7AtcBry6l/LnL2CRpomkT+R8HDi+lzOs6nsnIybv7oO2VXwBM7ToWSZIkLT+11icDs4CnAbfTzKl0DjDDpJIk9cXzgdcBX+06kMnKxFL/zMeKMEmSpEmj1roRTVJpY5o5N9cFPlFK2b+UMr/T4CRpgiqlzKFZHOFfaq0HdB3PZGRiqX/mY8WSJEnSpFBr3Ry4EHgKkMBC4IBSyqc6DUySJoe3AT8HPtwm+bUcmVjqH1vhJEmSJoFa61bAecAGNBXrvwdmllIu6DQwSZokSikP0rTDrQScWmud0nFIk4qJpf6xYkmSJGmCq7W+CPgOzefqL9GsALdjKeU3nQYmSZNMKeVG4FjgeYDVosuRcwD1j3MsSZIkTWC11hnAicD9wJ6llJs6DkmSJrsesBPwulrrWaWUy7sOaDKwYql/FmBiSZIkaUKqte4BfA2YC7zcpJIkda9doX0/4E7gS7XWNToOaVIwsdQ/tsJJkiRNQLXW/YHjgTuAXUopt3QbkSRpUCnlL8C7gacCJ3UbzeRgYql/TCxJkiRNMLXWQ4DPAbfSTND9x45DkiQNU0o5HzgdeGn7vq0+MrHUP7bCSZIkTSC11sOATwC/pEkq3dlxSJKkJXsncCPw/lrrtI5jmdBMLPWPFUuSJEkTRK31SOCDwHU07W/3dhuRJOnRlFLmA/vT5D1OqbVO6TikCcvEUv9YsSRJkjQB1Fp7wL8DVwC7llLmdRySJGkplFJ+BRwFbNX+qz4wsdQ/84GptdboOhBJkiQtu1pr1FqPAw4BvgfsXkp5oOOwJEnLoJRyLM17+P611hldxzMRmVjqn/tpWuFsh5MkSRpnaq0rAF+nWbb6HGCvUsqCbqOSJI3QAcBtwLG11id2HcxEY2Kpf+6jaYVbtetAJEmStPRqrSsC/wW8imZVof1KKYu6jUqSNFKllHuAdwDr0dw00CgysdQ/99FUK63WdSCSJElaOrXWlYCzgF2AE4CDSynZbVSSpMerlPI94BRg51rrO7qOZyIxsdQ/99AklqxYkiRJGgdqrasC5wHbAceUUv7dpJIkTSjvAa4HBmqtT+86mInCxFL/3IutcJIkSUstIj4eEXdGxIrt89kRMa39mT2K55kdEX+37HStdU3gQuD5wCdLKUeM1vkkSWNDKWUh8FpgEfCNWqtzIo8CE0v9cy9WLEmSJC2LlwO3ADv18ySZOT0zFw4+r7WuA1wMPAv4UCnl0/08vySpO6WU3wKfBJ4NHNNxOBOCiaX+uZumYsk5liRJkh5DRGwJ/BaowF6L2WTFiDghIi6PiIsiYuN2vz0i4rKIuDQiro2IGe34gRHx2Yg4KyKuiYijhpzrlvbf9ddbb73//sMf/vDbuXPnPv/b3/72NaWUL/X9YiVJnSqlHE9TpbpPrXW3ruMZ70ws9c99QAJP6DoQSWNTRGwaEedExI8i4uqIOHwZ9n2oLaT98vSGvgUqScvHXsDZ7c+eETH8c+rWwLGZuT1wBvCFdvzZwO6Z+RLgNcCHh+yzG7B/u+92EfHMoQfcbrvtpn/mM595xlOe8pR7V1999VJrXSUinjbqVyZJGosOBH4PHFVrXbvjWMY1E0v9cz8wH3hi14FIGnvaL0xnAd/JzK1pJoqdGRH7LuuxMvOkzPzqMpw7IqJGxCuX9VyS1EevAr6bmTfRTCmww7DXb8jM69vHpw15/TzgyxHxfeD/AU8dss+5mTk3MxP4Gc0y0wDUWp956KGHfmTRokXrHHXUUX/Zd9993wVsMWx/SdIEVUq5D3gL8CTglFprdBzSuGViqX/uBxZgxZKkxdsFWC0zTwTIzAeBw4CMiH0j4gcRcUVEHD04wWxEbNtWNp1Hc4eFdvzAiDiyfTw7Ig6JiFkR8ZOI2Lkdf1JEfDciLgOuA36QmWctzwuWpCWJiM2BTYCTI+ICmukEXjNss6Grsy2keb+cAlwAnJiZ2wGvAIZOyj1v2DGmAGy22WZTge/OnTt3g+OPP37OVVddtU1b8XT5sP0lSRNYKWXwpsQOwEDH4YxbJpb6Zx5WLElasi2AG4YOZOa1NEmfI4CXAtsCGwIHR8RU4GTgwMzcDbjzUY791MycAewLfLwdOwy4LjN3AApgtZKksWQvYCAzX56ZLwf2BvYEht49fsbgvEo0K/qcD6zR/sxpxw95rBPVWrd797vfvR7A0Ucffc1VV131X5l5f9smN31UrkaSNJ4cDvwYeGetdcuugxmPTCz1z2Ar3JpdByJpTFpAs8zpcDOBizLzvrZ14zRgV5qViuZm5k/b7c57lGN/E6BtJxls+7iDhyso13qcsUvSaNsdOHPwSWZeRbMQyouHbPM54Oi28vIVwLsz8x7gP4ArI2IWzXvdyks6ycEHH7w1cNr8+fMT+Jcbb7xxAHhnRFwKvK+NwXk2JGkSKaUsorkh+wBwUq11pY5DGndW7DqACWywFW6NrgORNCZdDxw6dCAiXjj4cNi2C3nkjYBH6wEf2vox2NLxXeCKiNgG+ANw8DJFK0l91E7IPXzsOcOGPrKEfT8DfGbI0Nfa8ZOGbnf66aefTTPh910bbrjhdqWUW0spAP888sglSRNBKeV/a60fAb4IfAl4Y8chjStWLPWPFUuSligz5wD3Dq7mFhErA5+gSUbvEhGrtZvuDZxDM+ns2hGxea31aWuttdaBy3jKdwP7ZeaLMvPVmfmrUbkQSRoHaq2vo/mi8CdgRinl1o5DkiSNMaWUrwPnAnvVWl/VdTzjiYmlPimlLKSpWLKMTtKS7AW8JiJ+RDM/yNmZ+UngKGB22+5xM3BSZj4AHBAR9YMf/OD1z3ve87ZbxnP9EDg+IuZExMUR8ZWI2GQ0L0aSxqJa69tp3ld/S5NUur3jkCRJY9cbaP5efLrWum7XwYwX0UzhoX6otf4M+EkpZZ+uY5E0/rVLoJ5BsyT3rFLKzKXdNyJ+BWydmXdHxIrAu4CNMvNd/YlWkrpXa30fzdxJPwV2K6XM7TgkSdIYV2vdhqZj4HrgZaUUkyaPwYql/poPTO06CEkTxodoVosDmFZrXeIEtYvxFeDCiLgEuKw9zpdHOT5JGjNqrR+nWennWuClJpUkSUujlHI1zWfnbYEPdhzOuODk3f21ABNLkkZBrXUn4C3A4NxLGwMvAS5cmv0z89PAp/sTnSSNHW1159HA62kS6XuWUh7sNipJ0jjzYWA74K211nNKKdd1HdBYZsVSf1mxJGm0bAZcB9wK3EdzY2C/TiOSpDGmTSp9hWaOjAuBV5pUkiQtq1LKImBf4K/AibXWVToOaUwzsdRf87EqTNIoKKWcAOxGk1S6HHg1zaoVkiSg1roCcArNaprfAUopZUG3UUmSxqtSyp+AI2hu8DqFxKMwsdRftsJJGk0bAxsAPyylnFVKqV0HJEljQa11ReDbwO40yaXXtXebJUkasVLKN4GzgT1qrS7KtQQmlvrLiiVJo2k6zRxL53cchySNGe1CBt8FdgaOL6W82RV8JEmj6E3AzcDHaq3rdx3MWGRiqb+cY0nSaJoB3EYz15IkTXq11sFk+4uAo0spAx2HJEmaYEopfwMOAlYHTmtbrzWEv5D+WoAVS5JGz8bAH9s/bpI0qdVanwBcBDwX+Ggp5chuI5IkTVTtqnBfArYBPtJxOGOOiaX+smJJ0qhoVzraEPhj17FIUtdqrU8CZgFPBw4vpXyu45AkSRPfx4ErgINrrdt0HcxYYmKpv6xYkjRaptEkln7ccRyS1Kla64bAxcBGwLtKKa7UI0nqu3b+vtcC9wBfbtuxhYmlfrNiSdJo2RlYGSfuljSJ1Vr/gab97cnAm0opp3YckiRpEiml/Bl4H81N3692G83YYWKpv+YDU53cS9IomE7TBvc/HcchSV06A1gTOKCUck7XwUiSJp9SypnAt4B/qbUe0HU8Y4FtWn3QJpLWABbR/I6fW2tN4DqXv5U0QhvRTNz9QNeBSFKH3gwsLKX8qOtAJEmT2tuAfwI+XGv9Xinl910H1CUrafpjd+CX7b/rA7NpMprRYUySxikn7pakRinlhyaVJEldK6U8CBxIM1XFqZO9S2lSX3wfnQ3cDKxLU7G0JnByKWVRp1FJGq82o0ks+WVKkiRJGgNKKT8FjgWeD3yy43A6ZWKpD9p2t/8D3N0O/QL4THcRSRrnZgArARd0HYgkSZKkh3wWmAMcVGvdrutgumJiqU9KKRcBV7dPLymlzOsyHknj2hzgeOCnXQciSZIkqdEWlewH3An8Z611jVrrvrXWl3Uc2nIVmc4l3S+11i1p5lfaspRye8fhSJIkSZKkUVZr3Q34BnAb8BTgzFLKQd1GtfxYsdRHbc/lziaVJEmSJEmasP4I3AU8G1gL2KDbcJavFbsOYKIrpfxP1zFIkiRJkqTRV2tdGzgHeOqQ4Sd1FE4nrFiSJEmSJEkagVLKX4B/Ak4Cbm2H1621TppCHudYkiRJkiRJepxqrU8FPg9MB7Yvpfy824iWDyuWJKkDEbFpRJwTET+KiKsj4vBl2HdaRMzuY3iSJqCIyIi4NCJmR8RVEfGdiFjzMfY5MiIOHKXzfzMinvrYW0qSND6VUv63lPKvwLbAn7qOZ3mZNKVZkjRWRMQKwFnAsZl5YkSsBJwfEbdk5mkdhydpYpuRmQsAIuIbwBto7qwuk4h4GfD2zNx9affJzH2W9TySJI1HpZRfdh3D8mTFkiQtf7sAq2XmiQCZ+SBwGJARsW9E/CAiroiIoyNiCkBEbNtWNp0HHDh4oIhYKSKOiYjvta8f0sH1SBpnImJlYH3g1+3zW4a8ttiqyIg4oK2y/CFwNPCqiFg9Ik5uK6F+2FYlrTB4zIj4t7Y66sltpdS0R9tHkiSNP/4RfxS9iE17Eef0In7Ui7i69xitKr0hH8qGjR/ZG2EZeS/i30eyn6QxbQvghqEDmXktcB1wBPBSmvLZDYGDI2IqcDJwYGbuBtw5ZNf3Ar/NzJ2B7YADI+LZ/b8ESePUrDZp9BvgD5l5ztLsFBFrAMcCMzPzhcD1wPbA5sC5mfmSdvxBmnklBm2ZmS/IzDuGjD3WPpIkjUm9iOxFXNqLuKQX8cNee1P38XznX8rzntSLmN6v4z9eJpaWoPdwq8p3BjK3pvnCNrMXse9yDuWdy/l8kvpvAbBoMeMzgYsy875sVlY4DdgVeBYwNzN/2m533pB99gD2ab8oXgisDphYkrQkMzJzOrAxsEZEvGsp93sA+Gu7zxRgcG6m64FNI2JORFwKvIS/X275m4s51mPtI0nSWDZjIHMnYAbw3p43dZ1j6VHsAqw20LaqDGQ+2Is4DHhGm1w6FAjgSmBgIHPh0J17ER8DXg78DpgL3NKOP4umfHxVmi+Wbx/IvLE99r/RfHD7Nc2cB6cCG/SaL4wfHsi8tK9XLGl5uZ7mPeQhEfHCwYfDtl3II28CDN1mCvCWzLxqVCOUNKFl5sKI+DrwNpo5lhZFRLRJ7VUWs/38iLgImEPz2ebszLw0It5GkxjaIzPvjoijaN6XBt23mNO/9TH2kSRpzBvIvK8XcQ2w1dDxXsQRwMtoPrMvBF4LPBN460DmXu02H6T5TjAH+AKwEc0N4k8OZJ7ZazoWvgj8I3Az8ITlclEjZMXSkj2iVWXgUVpVhm7Xi3g5TUn3i4F9af4nohexIk07yyEDzd3CdwMn9iLWAo4Eth7I3Aa4GNhsIPOVwG0DmdNNKkkTR2bOAe6NiDfAQ3OdfAJYA9glIlZrN90bOAf4GbB2RGzejr9myOEuAPZvjzM1Is6LiH9YDpchafzbhub9BZpk0eAd1zcO3zAiNgS2yMynZeb0zPxc+9JmwBVtgmh94FVLcd6R7CNJ0pjSa/42vgi4esjYmjTJpB0GMrcDTgLeBFwCPK8X8cR2011pPscfBZw50ExrsQvw8V7Ek2gKTZ4wkLktzQ2Zf1wuFzVCViwt2aO2qgxk3gfQizgNeD1w3JBtdgHOGGhXXelFXNyOP6P9+XovHio4eBJNRdP1wH+32142kHn96F6OpDFmL+D4iHgLzR+fUzPzmIi4A5gdEQ8AlwInZWZGxAHANyPiPuDcIcf5OHBsRFxB8551Qmb+ZvleiqRxZFZEJM1nwDtpPrgCfAg4OSLuBk4Bth623x00Ce4f0FRXzwO+DRwDnBYRM2mWVf4vYO3HiGEk+0iSNFbM6jV/S6HBxPByAAADdElEQVTpQPr14Pf7gcy5vYibgPN7EavTVBr9aCBzUS/iTOA1vYgbgJ8MZM7vRexB0xX1jvZ4ATydJqdwcnvMe3sRly+/y1t2JpaW7BGtKr1Hb1UZakltK1OA37fVSn+nF7E98ALgecBRvYhzBjI/OoK4JY0Dmfk7mjsVw8dPAE5YzPhs/v6L3mfb8XksprpAkobLzOGfX4a+djnNZ5BBJ7TjRwJExMHAtzPz8Pb5k4CfAGdkczd1ccecNuz59CFPF7uPJEnjwIzBIpLhehFb0Ux9M3Mg81e9iN1pbihDc+Pms8CNtEkjmhzBawYybx92nEebCmPMsRVuCQbaVpVe26rSG9aq0ntkq8pQlwCv6EWs0G63Wzv+M2ClXsQL2mNu1Yv4Ti9iU5qKp6sGMr8EfA54brvPvF7EOv25SkmSpKUyC9gmIi6LiDk0iwh8NTPv7jguSZLGkmnAzW1SaSpw0OALA5nXABsALxzIvLIdfmhai17Emr2IOW073SU0i/TQi1iXMb56qhVLj24v4PjekFaVgcxjem2rSm9Iq8rQnQYyz24rkK4E7gYub8fn9yL+FTimF7GIppT87QOZv+1F3Axc2R5zIU0fJTRL+17ei6gD7V1DSZKk5Skzb6aZDkCSJC3Z+cB+vYiraHIBZ/D3XQrf4u87nt5Jk3PYs33+qbad7njguF4z3cUdwPf6H/rIRbP4hyRJkiRJkvqlF3EGzaryN3cdy2iyYkmSJEmSJKlPehHb0cyvdPFESyqBFUuSJEmSJEkaISfvliRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiPx/gOBXVkR9+YoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1152x648 with 12 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(nrows=3, ncols=4, figsize=(16, 9))\n",
    "axes = axes.flatten()\n",
    "fc = ec = 'darkgrey'\n",
    "for s, (section, result) in enumerate(results.groupby('section')):\n",
    "    if s > 11:\n",
    "        continue\n",
    "        \n",
    "    df = result.drop('section', axis=1).apply(lambda x: x.map(word2id))\n",
    "    most_similar_idx = find_most_similar_analogy(df)\n",
    "    \n",
    "    best_analogy = result.iloc[most_similar_idx, :4].tolist()\n",
    "\n",
    "    analogy_idx = [words.index(word) for word in best_analogy]\n",
    "    best_analogy = [a.capitalize() for a in best_analogy]\n",
    "    \n",
    "    coords = pd.DataFrame(vectors2D[analogy_idx])  # xy array\n",
    "    \n",
    "    xlim, ylim = get_plot_lims(coords)\n",
    "    axes[s].set_xlim(xlim)\n",
    "    axes[s].set_ylim(ylim)\n",
    "\n",
    "    for i in [0, 2]:\n",
    "        axes[s].annotate(s=best_analogy[i], xy=coords.iloc[i+1], xytext=coords.iloc[i],\n",
    "                         arrowprops=dict(width=1,headwidth=5, headlength=5,\n",
    "                                         fc=fc, ec=ec, shrink=.1),\n",
    "                         fontsize=12)\n",
    "    \n",
    "        axes[s].annotate(best_analogy[i+1], xy=coords.iloc[i+1],\n",
    "                         xytext=coords.iloc[i+1],\n",
    "                         va='center', ha='center',\n",
    "                         fontsize=12, color='darkred' if i == 2 else 'k');\n",
    "\n",
    "    axes[s].axis('off')\n",
    "    title = ' '.join([s.capitalize()\n",
    "                      for s in section.split('-') if not s.startswith('gram')])\n",
    "    axes[s].set_title(title, fontsize=16)\n",
    "\n",
    "fig.tight_layout();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Resources\n",
    "\n",
    "- [Distributed representations of words and phrases and their compositionality](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)\n",
    "- [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf?)\n",
    "- [Sebastian Ruder's Blog](http://ruder.io/word-embeddings-1/)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "47px",
    "left": "38px",
    "right": "1340px",
    "top": "66.5px",
    "width": "457px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
